factual

Micro-batching only matters in training

The distinction between micro-batch and global batch is irrelevant in inference (a forward-only pass can be labeled arbitrarily and the pipeline bubble is naturally filled by running the next inference immediately), but matters in training because there is an optimal global batch size requiring a hard stop to accumulate all sequences before the backward pass.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“the difference between micro-batch and batch doesn’t matter at all in inference because you can just call it whatever you want. It only matters in training because there is an optimal batch size.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...