Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.
factualpending
Speaker
Wes RothEvidence Quote
“this model that they made it's 16 experts by 3.8 billion parameters so with 6.6 billion activated parameters so the effect of sort of model size this model outperforms a 7 billion model”
Created: 8/10/2026, 10:52:56 PM
Your notes
Loading notes...