normative
Models should be organic, not mathematically uniform
Future models should move from the current regular structure—where every expert is the same size and paths merge back together quickly—toward a more organic structure with specialized portions, like a brain has areas good at different things, where you add capacity to grow expertise where needed and let the model's connectivity adapt to the hardware's connectivity (dense within a chip, sparser across chips, minimal across metro areas, sending only the most salient representations over long-distance links).
normativepending
Speaker
Jeff DeanEvidence Quote
“this notion of having different specialized portions, sort of areas of a model of a brain that are good at different things. We have a little bit of that in Mixture of Experts models, but it's still very structured.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...