causal

Memory wall, not compute, blocks million-token context

The compute cost of long context (including the quadratic/linear attention term) has such a low slope that it is well-approximated as flat until millions of tokens; the binding constraints on very long context are memory bandwidth and memory capacity, and context lengths have plateaued around 100-200K because there is no good solution to this memory wall—sparse attention helps via its square-root term but going too sparse degrades quality.

causalpending

Speaker

Reiner Pope

Evidence Quote

“The primary things that limit you to really large contexts are memory bandwidth and memory capacity, which is exactly this effect.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...