factual
Prefill is compute-limited, decode is memory-bandwidth-limited
Output (decode) being charged roughly 3-5x more than input (prefill) indicates that prefill is compute-limited while decode is memory-bandwidth-limited: processing many tokens in one pass amortizes the weight memory fetch across the pass length, so the per-token memory cost falls with longer passes, revealing the labs are heavily memory-bandwidth bottlenecked during decode.
factualpending
Speaker
Reiner PopeEvidence Quote
“The result we want to work towards is that prefill is compute-limited and decode is memory bandwidth-limited.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...