factual
Parallelism strategy mirrors model architecture
The best parallelism strategies in practice—expert parallelism (experts on different GPUs) and pipeline parallelism (layers on different racks)—physically resemble the model architecture itself rather than being exotic; among the dimensions a model scales (layers, model dimension, DFF, experts), only cutting along layers and experts is profitable given typical model sizes.
factualpending
Speaker
Reiner PopeEvidence Quote
“Every single one of those numbers you can choose to cut along. If those numbers are big enough, it eventually becomes profitable to cut along there. We have selected two of them.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...