causal

Active params bounded by compute, total params by scale-up

Two distinct hardware limits govern MoE sizing: active parameters are limited by compute cost, while total parameters are limited by the scale-up domain size, which is why model parameter counts stalled until larger scale-up domains (e.g., Blackwell racks with tens of terabytes) made multi-trillion-parameter models plus their KV caches deployable.

causalpending

Speaker

Reiner Pope

Evidence Quote

“the active parameters, as we saw, are limited by the compute cost, and the total parameters are limited by the scale-up size”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...