factual
Frontier MoE models are highly sparse
Few modern MoE models are more than 10 percent dense, and frontier models activate closer to 1 percent of experts, so compute is already used fairly judiciously and is not the main source of waste.
factualpending
Speaker
Neil MovvaEvidence Quote
“I don't think that we're wasting too much on the MoE side.”
Created: 11/17/2026, 6:43:49 AM
Your notes
Loading notes...