factual

Per-hop rack latency stacks up in sequential decode

Each pipeline hop from rack to rack adds latency (Pope estimates on the order of a few milliseconds, possibly an order of magnitude off) traversing GPU to network card to top-of-rack or data-center switch and back; because decode is sequential, these hop latencies stack across pipeline stages, so a handful of stages can add ~10ms per token, a meaningful fraction when a token takes ~20ms.

factualpending

Speaker

Reiner Pope

Evidence Quote

“because it’s decode and sequential, they stack up across the stages. You can’t do them at the same time.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...