factual
Pipelining saves capacity not runtime in inference
In inference, pipeline parallelism is neutral for latency, memory time, and compute time—it merely moves memory time between racks—but it saves memory capacity per rack, letting you fit, say, a quarter of the model per rack instead of the whole model; its real benefits are larger in training.
factualpending
Speaker
Reiner PopeEvidence Quote
“In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We’re just moving the memory time from one chip to another chip... However, what we are saving on is memory capacity.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...