pretraining
The initial large-scale training phase where models learn general language patterns from massive unlabeled text corpora, before fine-tuning or RL on specific tasks
As defined in What does the next training paradigm look like?
Other definitions
The initial, very expensive phase of training a large model on vast data before fine-tuning or deployment.
Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]
Where it comes up · 8
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]4 claims
- Inside the Trillion-Dollar AI Buildout2 claims
- Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face2 claims
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development2 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job2 claims
- Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]2 claims
- Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]2 claims
- What does the next training paradigm look like?1 claim
Most important claims · 10 of 17Search all claims for “pretraining” →
- Pretraining on next-word prediction across trillions of tokens forces a model to store many concepts, and a second stage of RLHF on helpful, honest examples turns that base into an assistant, with the second stage using far less data and compute.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Pretraining a model to predict the next word across trillions of examples produces a model that is only moderately accurate (around 18 percent) but must store a large number of concepts internally, which is why it can predict next words in science papers or web text.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Proprietary data like Bloomberg's can be learned later by fine-tuning a general foundation model, because pretraining is the expensive and domain-agnostic step, so most companies will fine-tune rather than train from scratch.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Agents are pretrained to imitate human text, which gives them concepts such as sacrifice and collective, and then reinforcement learning on difficult rewarded tasks creates goal-oriented beings that creatively pursue goals much like ambitious humans, so talking about their motives is structurally justified.Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
- Although frontier model training is getting more expensive, the number of people who know how to train large models is rising and the second-comer discount is high, since later entrants can use known expertise and modern hardware and skip training intermediate models.Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]
- For Llama 3 70B, the KV cache grows about 320 KB per token of in-context learning, whereas pretraining stores about 0.075 bits per token, a roughly 35-million-fold difference in information stored per token between in-context learning and pretraining.Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job
- Most people should not be training general foundation models because entry now costs more than $0.5 billion to do pretraining of something interesting, making a general language model startup an easy way to lose money unless it is highly differentiated.Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]
- Frontier compute is supply-constrained because GPUs are booked one or two years in advance, and a senior Meta researcher demanded 10,000 H100s, which represents billions of dollars over five to ten years, so even funded startups must wait while incumbents move to the next generation.Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]
- Pretraining improvements yield very general gains in intelligence because of a deep link between compression and intelligence, so a pretraining-automation loop makes the model broadly better.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- The AI lab Magic announced it pretrained a model matching leading open-source base models such as DeepSeek V4 Pro for just half the compute used to train the original GPT-3, roughly 500,000 dollars today, and Patel thinks orders-of-magnitude further improvements are plausible.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development