factual

Bytes-per-token deduced from API pricing crossover

Setting the memory-time equals compute-time balance at 200K tokens and assuming ~100B active parameters lets you solve for KV cache bytes per token, yielding roughly 1,667 bytes (~2 kilobytes)—plausibly achievable via dense attention with shared global context across layers (e.g., 8 KV heads × d-head 128) as in Character AI and Gemma models, or via sparse attention.

factualpending

Speaker

Reiner Pope

Evidence Quote

“the number of bytes is typically the number of attention layers times two times d-head times the number of KV heads”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...