factual

Models are dramatically underparameterized

Contrary to the usual deep-learning narrative of overparameterization, models trained on the entire internet are dramatically underparameterized given task complexity, forcing them to compress many features via superposition and absorb noisy interference—and a bigger model is more sample-efficient because it has space for cleaner representations.

factualpending

Speaker

Trenton Bricken

Evidence Quote

“if you're trying to train a model on the entire internet and have it predict with incredible fidelity, you are in the underparameterized regime and you're having to compress a ton of things and take on a lot of noisy interference in doing so. When you have a bigger model, you can have cleaner representations to work with.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...