normative
Controlled experiments favor training from scratch
Despite the appeal of gluing models together, in the current period of rapid progress there is a countervailing scientific interest: to run controlled experiments comparing one thing to another (which helps decide what to build), it is often best to start from scratch and compare complete training runs—though versioned modularity, freezing a model and swapping in a variant module, could capture much of that benefit while making research cheaper and faster.
normativepending
Speaker
Noam ShazeerEvidence Quote
“it's often best to just start from scratch so you can compare one complete training run to another complete training run at the practical level because it helps us figure out what to build in the future. It's less exciting but does lead to rapid progress.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...