factual
Per-hop rack latency stacks up in sequential decode
Each pipeline hop from rack to rack adds latency (Pope estimates on the order of a few milliseconds, possibly an order of magnitude off) traversing GPU to network card to top-of-rack or data-center switch and back; because decode is sequential, these hop latencies stack across pipeline stages, so a handful of stages can add ~10ms per token, a meaningful fraction when a token takes ~20ms.
factualpending
Speaker
Reiner PopeEvidence Quote
“because it’s decode and sequential, they stack up across the stages. You can’t do them at the same time.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...