factual
Bytes-per-token deduced from API pricing crossover
Setting the memory-time equals compute-time balance at 200K tokens and assuming ~100B active parameters lets you solve for KV cache bytes per token, yielding roughly 1,667 bytes (~2 kilobytes)—plausibly achievable via dense attention with shared global context across layers (e.g., 8 KV heads × d-head 128) as in Character AI and Gemma models, or via sparse attention.
factualpending
Speaker
Reiner PopeEvidence Quote
“the number of bytes is typically the number of attention layers times two times d-head times the number of KV heads”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...