factual

Prefill is compute-limited, decode is memory-bandwidth-limited

Output (decode) being charged roughly 3-5x more than input (prefill) indicates that prefill is compute-limited while decode is memory-bandwidth-limited: processing many tokens in one pass amortizes the weight memory fetch across the pass length, so the per-token memory cost falls with longer passes, revealing the labs are heavily memory-bandwidth bottlenecked during decode.

factualpending

Speaker

Reiner Pope

Evidence Quote

“The result we want to work towards is that prefill is compute-limited and decode is memory bandwidth-limited.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...