factual

Pipeline scale-out beats scale-up by 8x-overcomeable factor

Pipeline parallelism across racks avoids a communication bottleneck because, although scale-out is 8x slower than scale-up, the scale-up MoE traffic is expanded by the number of activated experts times two (up and down) times layers per stage—a product that easily exceeds 8—so spending the slower scale-out only on single inter-stage token transfers is cheap relative to the all-to-all done inside each rack.

factualpending

Speaker

Reiner Pope

Evidence Quote

“There’s just a factor of 8 that we need to overcome. So we need the product of these three things to be bigger than 8.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...