causal
One rack bounds expert-layer size via all-to-all
A mixture-of-experts layer uses expert parallelism producing an all-to-all communication pattern that is a perfect fit for a single rack's full GPU-to-GPU connectivity; spanning two racks forces about half the tokens onto the ~8x slower scale-out network, making rack-to-rack a bottleneck, so one rack effectively bounds the size of an expert layer you can run.
causalpending
Speaker
Reiner PopeEvidence Quote
“one rack bounds the size of an expert layer you can do. This has been part of what’s been driving towards larger and larger interconnect domains.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...