causal
MoE and MLA drive DeepSeek's training efficiency
DeepSeek's training efficiency comes primarily from two techniques: a mixture of experts model (activating only a subset of parameters per token, dramatically reducing training and inference cost) and a new multi-head latent attention (MLA) technique using low-rank approximation to reduce memory usage.
causalpending
Speaker
Dylan PatelEvidence Quote
“those two main things are, one is they went to a mixture of experts model, which we’ll define in a second. And then the other thing is that they invented this new technique called MLA, latent attention.”
Created: 6/13/2026, 3:31:25 AM
Your notes
Loading notes...