factual

Sparsity buys huge param count for modest quality

Increasing sparsity in mixture-of-experts models yields diminishing returns on quality: empirically (Unified Scaling Laws for Routed Language Models) a 64-expert model with 370M active parameters matched a dense 1.3B model—a roughly 64x increase in total parameters for only about a 4x effective gain in quality-equivalent dense size.

factualpending

Speaker

Reiner Pope

Evidence Quote

“the 64-expert 370-million activated parameter model is as good as a dense 1.3-billion model”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...