factual

Mixture of Experts experts are interpretable

Contrary to claims that it is hard to decode what each expert does, in the original Mixture of Experts work the experts were relatively easy to understand—one expert handled words referring to cylindrical objects, another was good at dates and times—though you don't need that human understanding to operate the model, since a learned router selects experts based on the example.

factualpending

Speaker

Noam Shazeer

Evidence Quote

“this expert, was getting words referring to cylindrical objects... This one's super good at dates.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...