causal
Weight Updates Sacrifice In-Context Sample Efficiency
Moving learning into the weights sacrifices the sample efficiency of in-context learning, because gradient updates are extremely sample inefficient, which is why shipped online-learning models have all learned the same thing across millions of users.
causalpending
Speaker
Dwarkesh PatelEvidence Quote
“But the moment you move into the weights, you have to give up on in-context learning's sample efficiency. Because gradient updates are super sample-inefficient”
Source
Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:25:50 AM
Your notes
Loading notes...