Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.

factualpending

Speaker

Wes Roth

Evidence Quote

“this model that they made it's 16 experts by 3.8 billion parameters so with 6.6 billion activated parameters so the effect of sort of model size this model outperforms a 7 billion model”

Source

Microsoft Goes FULL NUCLEAR to Develop AGI...— Wes Roth
Created: 8/10/2026, 10:52:56 PM

Your notes

Loading notes...