context length
The amount of past information (tokens, observations, time) a model can attend to when producing an output.
As defined in Fully autonomous robots are much closer than you think – Sergey Levine
Where it comes up · 7
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served4 claims
- What does the next training paradigm look like?2 claims
- Fully autonomous robots are much closer than you think – Sergey Levine2 claims
- Sam Altman #22 claims
- Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"1 claim
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI1 claim
- Deepseek Dylan Patel Nathan Lambert1 claim
Most important claims · 10 of 13Search all claims for “context length” →
- The model can execute minute-long dexterous tasks with only about one second of context because the memory demand is an instance of Moravec's paradox: well-rehearsed physical skills are baked into the network and require little working memory, unlike cognitively demanding tasks that require holding many pieces in mind.Fully autonomous robots are much closer than you think – Sergey Levine
- Today's systems lack a middle ground between large-scale training (producing a single static snapshot model) and in-context learning (sample-efficient but destroyed each instance); a useful capability would combine online learning with cognitive skills like introspecting on one's own knowledge and actively seeking out knowledge to fill holes—partly unexplored because ever-longer context windows removed the incentive.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- For reasoning-model inference, memory (bandwidth and capacity) is the most important of the three GPU axes (FLOPS, memory, interconnect), because the KV cache grows quadratically with context length and reasoning models generate tens of thousands of tokens, creating memory pressure that limits how many users can be served simultaneously.Deepseek Dylan Patel Nathan Lambert
- If, as Dario claims, in-context learning rather than continual learning suffices for AGI, then matching a month of working with an employee requires roughly hundred-million-token context lengths, which would demand a major ML-infra change to memory bandwidth (unless sparse attention substitutes)—a constraint Pope sees no good path to solving given HBM stagnation.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Gemini 3.1 charging 50% more above 200K tokens likely reflects a two-tier pricing structure aligned with the crossover point where memory-fetch time overtakes the constant compute time as a function of context length; solving the balance with ~100B active params and ~2kbytes/token reproduces this, since labs price close to cost to avoid being undercut.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The compute cost of long context (including the quadratic/linear attention term) has such a low slope that it is well-approximated as flat until millions of tokens; the binding constraints on very long context are memory bandwidth and memory capacity, and context lengths have plateaued around 100-200K because there is no good solution to this memory wall—sparse attention helps via its square-root term but going too sparse degrades quality.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- When memory fetch is linear in context length (true for dense attention), running at twice the optimal context length can roughly halve model FLOPs utilization, because you push from the balanced compute-bound/memory-bound point into being memory-limited; sparse attention scales much better and avoids this.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Extending context length to 10 or 100 million tokens is an engineering and inference problem, not a research problem—you must store the entire KV cache and juggle memory across GPUs; degradation occurs when you train at a short context length but serve at a long one, so the solution is training at longer contexts and learning to serve them at inference.Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"
- Robots face a trilemma where inference speed, context length, and model size all consume more inference compute and oppose each other, with current systems at roughly 100ms inference, one second of context, and a couple billion parameters, each orders of magnitude below human-brain equivalents.Fully autonomous robots are much closer than you think – Sergey Levine
- The question of whether RLVR can generalize so strongly that spending trillions on RL environments could produce fully human-like general intelligence is an empirical question; Dario's quote that model performance degrades when serving at longer context lengths than trained suggests short-horizon RL training may not generalize to long-horizon performance.What does the next training paradigm look like?