causal

One rack bounds expert-layer size via all-to-all

A mixture-of-experts layer uses expert parallelism producing an all-to-all communication pattern that is a perfect fit for a single rack's full GPU-to-GPU connectivity; spanning two racks forces about half the tokens onto the ~8x slower scale-out network, making rack-to-rack a bottleneck, so one rack effectively bounds the size of an expert layer you can run.

causalpending

Speaker

Reiner Pope

Evidence Quote

“one rack bounds the size of an expert layer you can do. This has been part of what’s been driving towards larger and larger interconnect domains.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...