scale-up network
The fast intra-rack interconnect (e.g., NVLink) giving full GPU-to-GPU connectivity in about two hops.
As defined in Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Where it comes up · 1
Most important claims · 8 of 8Search all claims for “scale-up network” →
- Frontier labs doing inference are largely within a single scale-up domain: they maximize expert parallelism up to the scale-up domain size and do very little or no pipelining (DeepSeek reports doing a lot of expert parallelism), reserving some pipelining only for models too large to fit a rack's memory; tensor parallelism is no longer profitable because experts are now so small.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Larger scale-up domains massively reduce weight memory-fetch latency because all GPUs in a scale-up domain load weights in parallel, so effective bandwidth equals scale-up size times per-GPU bandwidth; this term grew ~8x from Hopper while per-GPU bandwidth only grows ~1.5-2x per generation, meaning scale-up size—not memory capacity—solves the bandwidth/latency problem.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The reason racks use a hierarchy of switches rather than one giant scale-up switch is cabling congestion: doubling GPUs per rack requires literally twice the cable density, and physical constraints—connector and backplane density, cable bend radius, rack weight, power, and cooling—are pushed to extreme limits, so cable space, not memory, bounds scale-up domain growth.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Pipeline parallelism across racks avoids a communication bottleneck because, although scale-out is 8x slower than scale-up, the scale-up MoE traffic is expanded by the number of activated experts times two (up and down) times layers per stage—a product that easily exceeds 8—so spending the slower scale-out only on single inter-stage token transfers is cheap relative to the all-to-all done inside each rack.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- A mixture-of-experts layer uses expert parallelism producing an all-to-all communication pattern that is a perfect fit for a single rack's full GPU-to-GPU connectivity; spanning two racks forces about half the tokens onto the ~8x slower scale-out network, making rack-to-rack a bottleneck, so one rack effectively bounds the size of an expert layer you can run.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The increase in scale-up domain size from Hopper (8 GPUs) to Blackwell (72) came mostly from a product decision to switch the form factor from trays to racks with no substantial technical barrier, whereas the jump to Rubin's ~500 involves some marketing math but at least a genuine 4x from a more complex rack design that runs more cables.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- In Nvidia's rack topology, GPUs talk within a rack via fast NVLink (scale-up network) reaching any GPU in two hops through central NV switches, while leaving the rack uses a scale-out network up to a data-center switch that is typically about eight times slower in bandwidth.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Two distinct hardware limits govern MoE sizing: active parameters are limited by compute cost, while total parameters are limited by the scale-up domain size, which is why model parameter counts stalled until larger scale-up domains (e.g., Blackwell racks with tens of terabytes) made multi-trillion-parameter models plus their KV caches deployable.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served