sparse attention
An attention mechanism that selects only the relevant tokens to attend to (via a lightweight indexer) rather than attending to all tokens, as in DeepSeek-V3.2.
As defined in Ai Sota #2026
Where it comes up · 4
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served4 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job2 claims
- What does the next training paradigm look like?1 claim
- Ai Sota #20261 claim
Most important claims · 8 of 8Search all claims for “sparse attention” →
- Long context is fundamentally a no-free-lunch trade-off: an RNN/Mamba single fixed state is cheap but forgets as context grows because it can't compress everything, while transformers remember every token (great for lookup) but are expensive due to the quadratically-growing KV cache and dot product—so progress comes from finding the right Goldilocks ratio of attention layers to compressed states.Ai Sota #2026
- The compute cost of long context (including the quadratic/linear attention term) has such a low slope that it is well-approximated as flat until millions of tokens; the binding constraints on very long context are memory bandwidth and memory capacity, and context lengths have plateaued around 100-200K because there is no good solution to this memory wall—sparse attention helps via its square-root term but going too sparse degrades quality.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- When memory fetch is linear in context length (true for dense attention), running at twice the optimal context length can roughly halve model FLOPs utilization, because you push from the balanced compute-bound/memory-bound point into being memory-limited; sparse attention scales much better and avoids this.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Some of the sparse attention mechanisms published by DeepSeek change the memory-fetch scaling so that a square root appears in the KV term, dramatically improving how context-length memory cost scales compared to dense attention.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Setting the memory-time equals compute-time balance at 200K tokens and assuming ~100B active parameters lets you solve for KV cache bytes per token, yielding roughly 1,667 bytes (~2 kilobytes)—plausibly achievable via dense attention with shared global context across layers (e.g., 8 KV heads × d-head 128) as in Character AI and Gemma models, or via sparse attention.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Architectural innovations like sparse attention and KV cache compaction could provide intermediate representations to solve the memory-scaling problem, and a new architectural innovation is proposed almost every week in recent research.What does the next training paradigm look like?
- Sample efficiency and continual learning are deeply connected because on-the-job data is scarce and learning from it requires sample efficiency; in-context fast weights from attention scale poorly in memory, so architectures need an intermediate representation.Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job
- Architecture does not seem to be the fundamental bottleneck for continual learning, since many working ideas such as sparse attention and KV cache compaction already exist, so the loss function may be the real bottleneck.Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job