MoE
Mixture of Experts — a sparse model architecture where only some sub-networks (experts) activate per input, requiring load balancing across them.
As defined in Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
Where it comes up · 7
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]3 claims
- Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat2 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]2 claims
- Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"1 claim
- Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute1 claim
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken1 claim
- Satya Nadella — How Microsoft is preparing for AGI1 claim
Most important claims · 10 of 11Search all claims for “MoE” →
- Unlike human brains, where storing and loading memory is cheap and math is expensive, chips find loading data very expensive and math very cheap, and math is getting cheaper faster than memory because of fundamental DRAM limits, so models should be designed to use far more compute.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Few modern MoE models are more than 10 percent dense, and frontier models activate closer to 1 percent of experts, so compute is already used fairly judiciously and is not the main source of waste.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Extending context length to 10 or 100 million tokens is an engineering and inference problem, not a research problem—you must store the entire KV cache and juggle memory across GPUs; degradation occurs when you train at a short context length but serve at a long one, so the solution is training at longer contexts and learning to serve them at inference.Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"
- Because semiconductors and deployment are so supply-limited, a GPU is not priced by the comparative new chip you could buy but by the value of work it can do today; since newer models like GPT-5.4 are cheaper to serve, higher quality, and have a far larger TAM than GPT-4, an H100 produces more valuable tokens and is therefore worth more today than three years ago.Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute
- Most of the advances in AI come from algorithm and computer-science improvements (MoE, attention mechanisms) rather than raw hardware, since Moore's Law gives only ~25%/year while computer science can yield 10x; therefore China's large army of AI researchers—forced by compute limits to invent smarter algorithms—is its fundamental advantage, and DeepSeek launching first on Huawei would be a horrible outcome for the US.Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat
- Infrastructure should not be optimized for one model architecture, because an MoE-like breakthrough could change the network topology and strand capital optimized for a single architecture; you must build a fleet capable of supporting multiple model families and lineages, and to be a serious hyperscaler you must stay open and support an ISV ecosystem rather than owning every category.Satya Nadella — How Microsoft is preparing for AGI
- The largest inefficiency is in attention memory use: the KV cache is quite uncompressed, storing many kilobytes per token and likely off by an order of magnitude or two, and its entropy is not earning its keep.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Because Moore's Law now yields only ~25% per year, the only way to achieve 10x-100x leaps is to change the algorithm and how it's computed yearly; Nvidia's general programmability via CUDA enables new architectures (hybrid SSMs, fused diffusion/autoregressive models, MoEs), which is why Blackwell achieved roughly 50x Hopper's efficiency despite only ~75% transistor improvement.Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat
- Models are moving toward dynamism, meaning per-token or per-user control of computation and memory in attention and mixture-of-experts routing, and current hardware built before these architectures has overhead that makes such dynamic computation inefficient.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- DeepSeek's good research taste manifests as designing models perfectly to their hardware constraints: they used MLA to trade flops for memory bandwidth on flop-rich H800s, then moved to more memory-bandwidth-oriented NSA as export controls reduced their chips, and improved their MoE load balancing from auxiliary losses to cleaner bias terms—iteratively co-designing algorithm and hardware in a simple, elegant way backed by great engineering.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken