Pre-training
The initial large-scale phase of training an AI model on vast web-scale text data before any task-specific tuning.
As defined in cwt_brendan-foody
Other definitions
The initial phase of training a large language model on massive unlabeled datasets to learn general language patterns and representations before fine-tuning on specific tasks.
Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad GerstnerThe initial phase of AI model development where the model learns general language patterns and knowledge from large amounts of unlabeled data across diverse domains
How Claude's AI Financial Analyst is Changing Investing (Insider Unveils)
Where it comes up · 33
- Ilya Sutskever – We're moving from the age of scaling to the age of research7 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research6 claims
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken6 claims
- Yann LeCun's $1B Bet Against LLMs [Part 1]6 claims
- Ai Sota #20266 claims
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history6 claims
- Capital, Compute & AI Scaling5 claims
- AI Trends 2025: Mary Meeker Deck Deep Dive Part 14 claims
and 25 more
Most important claims · 10 of 89Search all claims for “Pre-training” →
- Pre-training has gotten extremely expensive and scaling it implies serving a very large model, but the cost of serving millions of users (billions of dollars of compute) dwarfs the relatively low training cost (DeepSeek's ~$5M, OLMo 3's ~$2M cluster rental), so labs shrink models to reduce serving costs even though bigger models would be smarter.Ai Sota #2026
- Pre-training scaling laws are now being supplemented by post-training scaling, multimodality scaling, synthetic data generation, and inference-time scaling, with inference-time scaling now scaling up to achieve intelligence through reasoning, simulation, and reflection rather than just model size.Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner
- A single pre-training cluster cannot exceed roughly 200,000 Hoppers that are coherent, meaning all GPUs share memory and act as one system, so progress requires waiting for the next chip generation to scale further.Nvidia v. Google, Scaling Laws, and the Economics of AI
- Pre-training scaling is slowing down because 9-10 orders of magnitude of progress since AlexNet means most low-hanging fruit has been picked, requiring research breakthroughs like reasoning, synthetic data, tool use, and AI scaffolding to continue intelligence scaling.AI's Trillion-Dollar Opportunity: Sequoia AI Ascent 2025 Keynote
- The choice between scaling pre-training and inference scaling is an economic one: pre-training is a fixed cost giving the model a permanent capability, while inference scaling costs money per query forever, so whether to spend more on training depends on how many users you have and how long the model stays on the market.Ai Sota #2026
- James Martens' invention of the Hessian-free optimizer in 2010, which successfully trained a 10-layer neural network end-to-end without pre-training from scratch, was the pivotal moment that convinced Sutskever that deep neural networks were fundamentally powerful because a network with 10 layers is analogous to the human brain firing for 100 milliseconds, allowing perfect object recognition.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
- GPT-1's success with a two-stage training process—extensive self-supervised pre-training on 7,000 books followed by supervised fine-tuning on labeled language tasks—proved that self-supervised pre-training could break models free from dependence on human-labeled data at scale.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Pre-training appears to be reaching data saturation, requiring exploration of alternative approaches like modified pre-training recipes or reinforcement learning paradigms to continue generating progress.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Industries with proprietary data—pharmaceuticals, law, finance—will eventually hire frontier-lab talent to build in-house models on their data, which would be a major pre-training unlock currently unavailable because that data (e.g. clinical trials) can't be accessed; general-purpose LLMs aren't scratching the surface of what a specifically-trained LLM could do.Ai Sota #2026
- The reporting that AI progress had stalled was really that models were not improving relative to expectations or the amount of compute applied, not that models were not getting better, so labs will keep trying to find the unlock.Capital, Compute & AI Scaling