factual

Sparse attention adds a square root to KV cost

Some of the sparse attention mechanisms published by DeepSeek change the memory-fetch scaling so that a square root appears in the KV term, dramatically improving how context-length memory cost scales compared to dense attention.

factualpending

Speaker

Reiner Pope

Evidence Quote

“some of the DeepSeek papers that have published sparse attention end up putting a square root in this term”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...