causal

Bigger models are more sample-efficient via more circuits

No one has a good explanation for the scaling law with parameter count, but a sketchy account is that a bigger model is like a larger library of parallel, composable computations/circuits (analogous to mixture models or multiplicative weight updates with learned gating), so more width and depth give more chances that some circuit guesses correctly and gets upweighted, making bigger models more sample-efficient.

causalpending

Speaker

John Schulman (OpenAI Cofounder)

Evidence Quote

“a bigger model has a bigger library of different computations, including lots of stuff that's dormant and only being used some of the time, but it has more space to look for circuits to do something useful.”

Source

John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:30 AM

Your notes

Loading notes...