causal
Slightly exceeding optimal context halves MFU
When memory fetch is linear in context length (true for dense attention), running at twice the optimal context length can roughly halve model FLOPs utilization, because you push from the balanced compute-bound/memory-bound point into being memory-limited; sparse attention scales much better and avoids this.
causalpending
Speaker
Reiner PopeEvidence Quote
“I’m modeling the memory fetch as linear in context length. That depends on model architecture. It is true for all of the model architectures with dense attention. Sparse attention actually scales much better than that.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...