Mixture of experts
An AI model architecture (MoE) where only a subset of specialized sub-networks ('experts') activate per input, enabling very large parameter counts efficiently.
As defined in Lex Fridman Podcast #494 - Jensen Huang
Other definitions
A model design where the feedforward layer is split into many sub-networks ('experts'), and a router activates only a few per token, so the model can hold more knowledge without using all of it on every forward pass (a 'sparse' architecture).
Ai Sota #2026A model architecture where several specialized sub-networks exist and a router sends each token to a subset of them, so only part of the model is active per token.
Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]
Where it comes up · 15
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI5 claims
- The Engineering Unlocks Behind DeepSeek | YC Decoded4 claims
- Microsoft Goes FULL NUCLEAR to Develop AGI...4 claims
- Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]3 claims
- Deepseek Dylan Patel Nathan Lambert3 claims
- Mark Cuban3 claims
- This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)2 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served2 claims
and 7 more
Most important claims · 10 of 34Search all claims for “Mixture of experts” →
- Because AI model architectures change about every six months while hardware and system architectures change about every three years, NVIDIA must anticipate two to three years ahead by doing its own research, working with every AI company in the world, and keeping CUDA flexible enough to adapt—exemplified by building NVLink 72 to fit a 4-10 trillion parameter mixture-of-experts model in one computing domain.Lex Fridman Podcast #494 - Jensen Huang
- There is a misconception that Mixture of Experts means you avoid retrieving unused expert weights; in reality, because efficient serving uses very large batches of independent requests, all experts are looked at every step, with only a small fraction of the batch sent through each expert—so you still need the whole model in memory (enough HBM to hold it), and running an unused expert at batch size one would be massively inefficient given modern hardware's high operational intensity.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- DeepSeek's training efficiency comes primarily from two techniques: a mixture of experts model (activating only a subset of parameters per token, dramatically reducing training and inference cost) and a new multi-head latent attention (MLA) technique using low-rank approximation to reduce memory usage.Deepseek Dylan Patel Nathan Lambert
- As models become more efficient and cheaper to train, including locally on a PC or phone, there will be millions of specialized AI models rather than four or five dominant foundation models — assembled via open source, copy-paste, or mixture-of-experts meta front ends — because brands like Mayo Clinic and Harvard Medical School will license or keep their own IP rather than donating it to ChatGPT or Gemini.Mark Cuban
- Cursor is in a strong position because of its vast user data; their Composer model is a fine-tune of a large Chinese Mixture-of-Experts model (detectable because it sometimes responds in Chinese), and they update the model weights every 90 minutes based on real-world feedback from people using it—the closest thing to real-world RL happening on a deployed model.Ai Sota #2026
- A mixture-of-experts layer uses expert parallelism producing an all-to-all communication pattern that is a perfect fit for a single rack's full GPU-to-GPU connectivity; spanning two racks forces about half the tokens onto the ~8x slower scale-out network, making rack-to-rack a bottleneck, so one rack effectively bounds the size of an expert layer you can run.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- From GPT-2 to today's models, the architectures are fundamentally the same; the differences—Mixture of Experts, multi-head latent attention, group query attention, sliding window attention, RMSNorm vs LayerNorm—are tweaks and knobs, not fundamental changes, so you can convert one into another by adding components in a continuous lineage.Ai Sota #2026
- Grok Moe is a mixture of experts model with 16 experts and 3.8 billion total parameters, with 6.6 billion activated parameters, that outperforms a 7 billion parameter dense model and matches the performance of a 14 billion parameter dense model on the same training data.Microsoft Goes FULL NUCLEAR to Develop AGI...
- Contrary to claims that it is hard to decode what each expert does, in the original Mixture of Experts work the experts were relatively easy to understand—one expert handled words referring to cylindrical objects, another was good at dates and times—though you don't need that human understanding to operate the model, since a learned router selects experts based on the example.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- The core self-attention transformer architecture has not fundamentally changed in six or seven years, apart from becoming mixture-of-experts, and fixing its quadratic attention cost would add costs elsewhere, so innovation will likely come from data and auxiliary layers rather than a new transformer.Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]