factual
Pipeline scale-out beats scale-up by 8x-overcomeable factor
Pipeline parallelism across racks avoids a communication bottleneck because, although scale-out is 8x slower than scale-up, the scale-up MoE traffic is expanded by the number of activated experts times two (up and down) times layers per stage—a product that easily exceeds 8—so spending the slower scale-out only on single inter-stage token transfers is cheap relative to the all-to-all done inside each rack.
factualpending
Speaker
Reiner PopeEvidence Quote
“There’s just a factor of 8 that we need to overcome. So we need the product of these three things to be bigger than 8.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...