causal

Weight Updates Sacrifice In-Context Sample Efficiency

Moving learning into the weights sacrifices the sample efficiency of in-context learning, because gradient updates are extremely sample inefficient, which is why shipped online-learning models have all learned the same thing across millions of users.

causalpending

Speaker

Dwarkesh Patel

Evidence Quote

“But the moment you move into the weights, you have to give up on in-context learning's sample efficiency. Because gradient updates are super sample-inefficient”

Source

Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:25:50 AM

Your notes

Loading notes...