factual
Micro-batching only matters in training
The distinction between micro-batch and global batch is irrelevant in inference (a forward-only pass can be labeled arbitrarily and the pipeline bubble is naturally filled by running the next inference immediately), but matters in training because there is an optimal global batch size requiring a hard stop to accumulate all sequences before the backward pass.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“the difference between micro-batch and batch doesn’t matter at all in inference because you can just call it whatever you want. It only matters in training because there is an optimal batch size.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...