factual
Sparse attention adds a square root to KV cost
Some of the sparse attention mechanisms published by DeepSeek change the memory-fetch scaling so that a square root appears in the KV term, dramatically improving how context-length memory cost scales compared to dense attention.
factualpending
Speaker
Reiner PopeEvidence Quote
“some of the DeepSeek papers that have published sparse attention end up putting a square root in this term”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...