DeepSeek V3 uses a mixture of experts architecture with 671 billion total parameters but only 37 billion parameters activated for each token prediction, activating 11x fewer parameters per forward pass than Llama 3's 405 billion non-sparse model

factualpending

Speaker

Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

Evidence Quote

“deep seek V3 has 671 billion modern parameters but only 37 billion are activated for a given token prediction by contrast the largest and most capable Lama 3 Model doesn't use a mixture of expert architecture so it activates its full 405 billion for each token prediction in other words V3 activates 11x fewer parameters for each forward pass saving tons of computation”

Source

The Engineering Unlocks Behind DeepSeek | YC Decoded— Y Combinator
Created: 8/11/2026, 6:44:59 AM

Your notes

Loading notes...