causal

MoE and MLA drive DeepSeek's training efficiency

DeepSeek's training efficiency comes primarily from two techniques: a mixture of experts model (activating only a subset of parameters per token, dramatically reducing training and inference cost) and a new multi-head latent attention (MLA) technique using low-rank approximation to reduce memory usage.

causalpending

Speaker

Dylan Patel

Evidence Quote

“those two main things are, one is they went to a mixture of experts model, which we’ll define in a second. And then the other thing is that they invented this new technique called MLA, latent attention.”

Source

Deepseek Dylan Patel Nathan Lambert— Lex Fridman Podcast
Created: 6/13/2026, 3:31:25 AM

Your notes

Loading notes...