activated parameters
In mixture of experts models, the number of parameters in the expert(s) that are actually used to process a given input, as opposed to total parameters in all experts
As defined in Microsoft Goes FULL NUCLEAR to Develop AGI...
Where it comes up · 1
Most important claims · 1 of 1Search all claims for “activated parameters” →
- Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.Microsoft Goes FULL NUCLEAR to Develop AGI...