dense model
A neural network architecture where all parameters are active for every input, as opposed to sparse or mixture-of-experts models where only a subset of parameters are used per input
As defined in Microsoft Goes FULL NUCLEAR to Develop AGI...
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “dense model” →
- Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.Microsoft Goes FULL NUCLEAR to Develop AGI...
- Mixture of experts models scale more efficiently than dense models, solving training issues specific to mixture of experts architectures through gradient-informed approaches.Microsoft Goes FULL NUCLEAR to Develop AGI...