factual
MoE requires whole model in memory anyway
There is a misconception that Mixture of Experts means you avoid retrieving unused expert weights; in reality, because efficient serving uses very large batches of independent requests, all experts are looked at every step, with only a small fraction of the batch sent through each expert—so you still need the whole model in memory (enough HBM to hold it), and running an unused expert at batch size one would be massively inefficient given modern hardware's high operational intensity.
factualpending
Speaker
Noam ShazeerEvidence Quote
“It's that you are looking at all the experts, but you only have to send a small fraction of the batch through each one.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...