factual
Models are dramatically underparameterized
Contrary to the usual deep-learning narrative of overparameterization, models trained on the entire internet are dramatically underparameterized given task complexity, forcing them to compress many features via superposition and absorb noisy interference—and a bigger model is more sample-efficient because it has space for cleaner representations.
factualpending
Speaker
Trenton BrickenEvidence Quote
“if you're trying to train a model on the entire internet and have it predict with incredible fidelity, you are in the underparameterized regime and you're having to compress a ton of things and take on a lot of noisy interference in doing so. When you have a bigger model, you can have cleaner representations to work with.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...