base model
The pretrained LLM that does pure imitation learning before any RL fine-tuning.
As defined in Some thoughts on the Sutton interview
Other definitions
A foundational large language model trained on broad internet data before any task-specific fine-tuning or specialization
Perplexity CEO: Disrupting Google Search with AIA foundational large language model trained primarily on next-token prediction (general language understanding) without specialized instruction-following or reasoning training; serves as the foundation for specialized variants
The Engineering Unlocks Behind DeepSeek | YC Decoded
Where it comes up · 9
- OpenAI CEO Sam Altman | AI for the Next Era5 claims
- The Engineering Unlocks Behind DeepSeek | YC Decoded2 claims
- The Rise of Artificial Intelligence | AngelList Confidential 20222 claims
- Perplexity CEO: Disrupting Google Search with AI1 claim
- AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo1 claim
- Joe Carlsmith — Preventing an AI takeover1 claim
- Some thoughts on the Sutton interview1 claim
- AI, Employment, and Education (with Tyler Cowen)1 claim
and 1 more
Most important claims · 10 of 15Search all claims for “base model” →
- Deep Seek V3 was released in December as a general-purpose base model and Deep Seek R1 was released at the end of January as a reasoning model built on top of V3, not as entirely separate models.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Fine-tuning is more efficient than training from scratch for domain-specific models (like an AI lawyer) because it starts with a base model that already has world knowledge and reasoning ability, requiring only targeted data to push the model toward specialized behavior.The Rise of Artificial Intelligence | AngelList Confidential 2022
- Alignment researchers report that when red-teaming a base model that has not yet been RLHF'd—a raw next-token predictor—it is oddly hard to get it to help with things like making a bomb, with the model trying hard to refuse even before reward-based fine-tuning.Joe Carlsmith — Preventing an AI takeover
- Evolution does meta-RL to make an RL agent that can selectively do imitation learning, whereas with LLMs we go the opposite way: first building a base model that does pure imitation learning, then doing RL on it to make it a coherent agent with goals and self-awareness.Some thoughts on the Sutton interview
- Fine-tuning large language models on domain-specific data degrades their generality and ability to engage in open-ended conversation, so enterprise applications should use prompt engineering and in-context learning with base models rather than fine-tuning, which trades away the 'magic' of generalityPerplexity CEO: Disrupting Google Search with AI
- To create enduring differentiated businesses on large language model APIs, companies should focus on a 'middle layer' of tuning existing base models with specialized data flywheels rather than training models from scratch, as base model creation will consolidate to a few providers.OpenAI CEO Sam Altman | AI for the Next Era
- The world's chip manufacturing capacity constrains global LLM development such that no significant number of startups can compete on base model training due to physical constraints on available GPUs/TPUs, consolidating development to a few well-funded players.OpenAI CEO Sam Altman | AI for the Next Era
- Base models need to learn general capabilities including basic reasoning, world knowledge, and fundamental concepts, while specialized/fine-tuned models need to learn domain-specific knowledge (e.g., case law, legal procedures, document drafting conventions) that enables them to perform at expert level in specific domains.The Rise of Artificial Intelligence | AngelList Confidential 2022
- Most EdTech startups that modify base AI models with bells and whistles—delaying answers or walking through learning steps—will not succeed, because people will just use the basic foundation model they are used to; the equilibrium is one base model people work with directly, not specialized education wrappers.AI, Employment, and Education (with Tyler Cowen)
- Just as AI capabilities are now measured by 'time horizon' (tasks lasting minutes vs hours), human writers may have differing time horizons: many can write a great comment or three-paragraph Tumblr post but can't sustain coherence across a full blog post, and current AI's ~1-hour task horizon likely explains why it can't yet plan and execute a research-heavy blog post despite strong sentence-level style.AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo