pipeline parallelism
Splitting a model's layers across different racks so each rack processes a different stage of the forward pass.
As defined in Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Other definitions
A distributed training technique where different stages of a neural network are placed on different devices, with data flowing through the pipeline in stages to improve training efficiency.
Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner
Where it comes up · 2
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served8 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]2 claims
Most important claims · 10 of 10Search all claims for “pipeline parallelism” →
- Pipelining cannot reduce per-GPU KV cache memory: while fewer layers per rack means storing fewer layers of KVs, keeping all racks busy requires proportionally more sequences in flight, so the number of micro-batches equals the number of pipeline stages and the pipeline-stage factor exactly cancels—the weight footprint shrinks with pipelining but the activation/KV footprint stays constant.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Frontier labs doing inference are largely within a single scale-up domain: they maximize expert parallelism up to the scale-up domain size and do very little or no pipelining (DeepSeek reports doing a lot of expert parallelism), reserving some pipelining only for models too large to fit a rack's memory; tensor parallelism is no longer profitable because experts are now so small.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Pipeline parallelism across racks avoids a communication bottleneck because, although scale-out is 8x slower than scale-up, the scale-up MoE traffic is expanded by the number of activated experts times two (up and down) times layers per stage—a product that easily exceeds 8—so spending the slower scale-out only on single inter-stage token transfers is cheap relative to the all-to-all done inside each rack.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The best parallelism strategies in practice—expert parallelism (experts on different GPUs) and pipeline parallelism (layers on different racks)—physically resemble the model architecture itself rather than being exotic; among the dimensions a model scales (layers, model dimension, DFF, experts), only cutting along layers and experts is profitable given typical model sizes.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The distinction between micro-batch and global batch is irrelevant in inference (a forward-only pass can be labeled arbitrarily and the pipeline bubble is naturally filled by running the next inference immediately), but matters in training because there is an optimal global batch size requiring a hard stop to accumulate all sequences before the backward pass.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- In inference, pipeline parallelism is neutral for latency, memory time, and compute time—it merely moves memory time between racks—but it saves memory capacity per rack, letting you fit, say, a quarter of the model per rack instead of the whole model; its real benefits are larger in training.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Sharding a large matrix multiply across up to eight GPUs with NVLink reduces latency but scales sublinearly, so eight times the hardware yields perhaps 4 to 5 times the speed due to communication overhead, making tensor-parallel sharding a poor choice for throughput-optimized serving.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- A Blackwell rack has many tens of terabytes of memory, far exceeding the ~1 terabyte needed for a trillion-parameter model, so even in inference pipelining is rarely used because it reduces a memory-capacity number that is already comfortably small—implying hardware could have been designed with less HBM per GPU.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Each pipeline hop from rack to rack adds latency (Pope estimates on the order of a few milliseconds, possibly an order of magnitude off) traversing GPU to network card to top-of-rack or data-center switch and back; because decode is sequential, these hop latencies stack across pipeline stages, so a handful of stages can add ~10ms per token, a meaningful fraction when a token takes ~20ms.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- NVLink is mandatory for low-latency inference and tensor parallelism is basically Nvidia-only, so Sail does not care much about low latency and instead optimizes flops per dollar using other vendors' chips with alternative parallelism such as expert or pipeline parallelism.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]