← Jargon

Jargontechnology· 34 claims· 15 episodes

mixture of experts

An AI model architecture (MoE) where only a subset of specialized sub-networks ('experts') activate per input, enabling very large parameter counts efficiently.

As defined in Lex Fridman Podcast #494 - Jensen Huang


Other definitions
  • A model design where the feedforward layer is split into many sub-networks ('experts'), and a router activates only a few per token, so the model can hold more knowledge without using all of it on every forward pass (a 'sparse' architecture).

    Ai Sota #2026
  • A model architecture where several specialized sub-networks exist and a router sends each token to a subset of them, so only part of the model is active per token.

    Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]

Where it comes up · 15

and 7 more