in-context learning
The popular notion that LLMs learn from examples within a prompt; Chollet argues it is actually fetching of rule sets, not genuine learning.
As defined in ms_280
Other definitions
A capability of large language models where they learn to perform new tasks based on examples provided in the prompt (context) without explicit fine-tuning. The model uses patterns from demonstrations to adapt its behavior to new tasks.
Jeff Dean (Google): Exciting Trends in Machine LearningA large language model's ability to perform a new task from instructions or examples supplied in the prompt, without retraining its weights.
Amjad Masad - The Future of Software Creation - [Invest Like the Best, EP.310]
Where it comes up · 21
- 363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins4 claims
- Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"3 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job3 claims
- Breakthrough potential of AI | Sam Altman | MIT 20233 claims
- Andrej Karpathy — AGI is still a decade away2 claims
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development2 claims
- What does the next training paradigm look like?1 claim
- An audio version of my blog post, Thoughts on AI progress (Dec 2025)1 claim
and 13 more
Most important claims · 10 of 32Search all claims for “in-context learning” →
- GPT-3's in-context learning, now called prompting, means a single general language model can be directed to translation, sentiment analysis, email writing, or copy generation through natural-language instructions without retraining, so models are programmed by language rather than built for one task.Amjad Masad - The Future of Software Creation - [Invest Like the Best, EP.310]
- In-context learning, where a model infers a task from a few examples such as the Marie Curie completion, is a general routine that arises from prediction training, works after weights are frozen, and resembles the rapid, novel pattern-grokking attributed to System 2.363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins
- In-context learning happens after training ends and the weights are frozen, because the activations change as the model processes the prompt, so learning-like computation is not confined to the training phase.363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins
- Information memorized in model parameters is memory-efficient (roughly one fact per parameter) but squishy, leading to hallucination and confused facts because trillions of tokens are churned together; information in the context window is sharp and clear thanks to the attention mechanism, but each token costs far more memory (kilobytes to a megabyte of keys and values per token across layers), creating a fundamental trade-off.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- To turn a general language model into a specific product, first teach it the desired answer style and format through in-context examples, which does not change the model, and then optionally fine-tune it on proprietary data such as user questions, responses, and internal design documents.Jeremiah Lowin: Explaining the New AI Paradigm - [Invest Like the Best, EP.307]
- There are two paths to fast adaptation: continual learning (continuously updating model weights so it adapts to new information rapidly, like an employee who learns from feedback) versus in-context learning (loading a huge context window with extra information each prompt); Lambert is more bullish on simply providing very good context, since updating weights per person is too expensive.Ai Sota #2026
- Multimodal models sometimes improve performance on all domains by providing additional context and understanding, though models trained on highly specific targeted datasets often perform well on narrow problems; for complicated or data-constrained problems, large multimodal base models with small amounts of fine-tuning data are preferable.Jeff Dean (Google): Exciting Trends in Machine Learning
- Today's systems lack a middle ground between large-scale training (producing a single static snapshot model) and in-context learning (sample-efficient but destroyed each instance); a useful capability would combine online learning with cognitive skills like introspecting on one's own knowledge and actively seeking out knowledge to fill holes—partly unexplored because ever-longer context windows removed the incentive.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Simulation works for human pilots because they are intensely goal-directed toward flying the real airplane, whereas models trained on multi-domain data don't know they must solve a particular downstream task; the key to leveraging simulation is therefore building a foundation model good enough to have emergent ability to use auxiliary data, which itself requires the right objective grounded in real data.Fully autonomous robots are much closer than you think – Sergey Levine
- LLMs approximate Bayesian posteriors not by explicit factorized hypothesis updating but by amortizing inference through long learning, mapping novel contexts into Bayesian-like approximators in their activations after weights are frozen.363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins