batch size
The number of independent inputs processed together; large batch sizes are needed for efficient inference on MoE models.
As defined in Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Where it comes up · 2
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served7 claims
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI1 claim
Most important claims · 8 of 8Search all claims for “batch size” →
- There is a misconception that Mixture of Experts means you avoid retrieving unused expert weights; in reality, because efficient serving uses very large batches of independent requests, all experts are looked at every step, with only a small fraction of the batch sent through each expert—so you still need the whole model in memory (enough HBM to hold it), and running an unused expert at batch size one would be massively inefficient given modern hardware's high operational intensity.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- Setting weight-fetch time equal to weight-multiply time yields a hardware constant (FLOPs/byte ratio) of roughly 300 across recent GPUs, so the optimal batch size is approximately 300 times the model's sparsity ratio—e.g., DeepSeek activating 32 of 256 experts gives a sparsity of 8, implying a batch around 2,000-3,000 tokens; this depends only on sparsity, not model scale.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- There is a fundamental tension in training batch size: from an ML convergence-rate perspective smaller is always better because the gradient descent step uses the freshest information, while from a systems perspective smaller is worse; the optimal batch size is the trade-off between these two forces.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- From a pure bandwidth-time perspective, increasing sparsity is a win as long as you have enough users to fill larger batches, because the doubled weight-memory fetches are amortized away by running a bigger batch; the binding cost is instead the additional memory capacity the larger total parameter count consumes.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Batching many users together in a single forward pass is the dominant lever on inference cost: if you do not batch, the per-token economics can be roughly a thousand times worse than if you batch many users together, because the cost of fetching weights from memory is amortized across all batch elements.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Converting batch size to throughput (batch times ~60-64 per second) gives roughly 128,000 tokens per second for an optimal-batch system, which is only about one-thousandth of Gemini's reported hundreds of millions of tokens per second worldwide—implying batching provides only a modest push toward centralization.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Inference runs on a fixed schedule like a departing train: a new batch starts every ~20 milliseconds regardless of fullness; ready requests board, full trains leave waiting passengers for the next one, so worst-case queuing latency is one train interval plus one completion—about 40 milliseconds.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The distinction between micro-batch and global batch is irrelevant in inference (a forward-only pass can be labeled arbitrarily and the pipeline bubble is naturally filled by running the next inference immediately), but matters in training because there is an optimal global batch size requiring a hard stop to accumulate all sequences before the backward pass.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served