causal
Batching can improve inference economics 1000x
Batching many users together in a single forward pass is the dominant lever on inference cost: if you do not batch, the per-token economics can be roughly a thousand times worse than if you batch many users together, because the cost of fetching weights from memory is amortized across all batch elements.
causalpending
Speaker
Reiner PopeEvidence Quote
“if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...