factual
Long context is engineering, not research, problem
Extending context length to 10 or 100 million tokens is an engineering and inference problem, not a research problem—you must store the entire KV cache and juggle memory across GPUs; degradation occurs when you train at a short context length but serve at a long one, so the solution is training at longer contexts and learning to serve them at inference.
factualpending
Speaker
Dario AmodeiEvidence Quote
“There's the context length you train at and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.”
Source
Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:32:46 AM
Your notes
Loading notes...