normative

Models should be organic, not mathematically uniform

Future models should move from the current regular structure—where every expert is the same size and paths merge back together quickly—toward a more organic structure with specialized portions, like a brain has areas good at different things, where you add capacity to grow expertise where needed and let the model's connectivity adapt to the hardware's connectivity (dense within a chip, sparser across chips, minimal across metro areas, sending only the most salient representations over long-distance links).

normativepending

Speaker

Jeff Dean

Evidence Quote

“this notion of having different specialized portions, sort of areas of a model of a brain that are good at different things. We have a little bit of that in Mixture of Experts models, but it's still very structured.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...