forecast

Modularity enables parallel, continual model development

If model pieces could be developed somewhat independently—small teams creating great training data and training a modular piece (e.g., for Southeast Asian languages or Haskell reasoning) that hooks onto a larger model—then 100 teams around Google or people worldwide could collectively and continually improve one model, decomposing the current monolithic pre-training process and giving a software-engineering benefit, which is itself a form of continual learning.

forecastpending

Speaker

Jeff Dean

Evidence Quote

“you could have 100 teams around Google. You could have people all around the world working to improve languages they care about or particular problems they care about and all collectively work on improving the model. And that's kind of a form of continual learning.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...