Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargonmachine learning· 2 claims· 1 episode

dense model

A neural network architecture where all parameters are active for every input, as opposed to sparse or mixture-of-experts models where only a subset of parameters are used per input

As defined in Microsoft Goes FULL NUCLEAR to Develop AGI...


Where it comes up · 1
  • Microsoft Goes FULL NUCLEAR to Develop AGI...2 claims

Most important claims · 2 of 2Search all claims for “dense model” →
  • Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.Microsoft Goes FULL NUCLEAR to Develop AGI...
  • Mixture of experts models scale more efficiently than dense models, solving training issues specific to mixture of experts architectures through gradient-informed approaches.Microsoft Goes FULL NUCLEAR to Develop AGI...