causal

Batching can improve inference economics 1000x

Batching many users together in a single forward pass is the dominant lever on inference cost: if you do not batch, the per-token economics can be roughly a thousand times worse than if you batch many users together, because the cost of fetching weights from memory is amortized across all batch elements.

causalpending

Speaker

Reiner Pope

Evidence Quote

“if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...