causal
Active params bounded by compute, total params by scale-up
Two distinct hardware limits govern MoE sizing: active parameters are limited by compute cost, while total parameters are limited by the scale-up domain size, which is why model parameter counts stalled until larger scale-up domains (e.g., Blackwell racks with tens of terabytes) made multi-trillion-parameter models plus their KV caches deployable.
causalpending
Speaker
Reiner PopeEvidence Quote
“the active parameters, as we saw, are limited by the compute cost, and the total parameters are limited by the scale-up size”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...