dense attention
Attention in which each token attends to all previous tokens, making memory fetch scale linearly with context length.
As defined in Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Where it comes up · 1
Most important claims · 3 of 3Search all claims for “dense attention” →
- Increasing sparsity in mixture-of-experts models yields diminishing returns on quality: empirically (Unified Scaling Laws for Routed Language Models) a 64-expert model with 370M active parameters matched a dense 1.3B model—a roughly 64x increase in total parameters for only about a 4x effective gain in quality-equivalent dense size.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- When memory fetch is linear in context length (true for dense attention), running at twice the optimal context length can roughly halve model FLOPs utilization, because you push from the balanced compute-bound/memory-bound point into being memory-limited; sparse attention scales much better and avoids this.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Setting the memory-time equals compute-time balance at 200K tokens and assuming ~100B active parameters lets you solve for KV cache bytes per token, yielding roughly 1,667 bytes (~2 kilobytes)—plausibly achievable via dense attention with shared global context across layers (e.g., 8 KV heads × d-head 128) as in Character AI and Gemma models, or via sparse attention.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served