factual

In-context learning may run internal gradient descent

In-context learning is pattern completion within a token window that emerges from pre-training, and there is evidence (e.g. a linear-regression-in-context paper) that it may internally run a small gradient-descent-like loop in the layers of the network rather than being a fundamentally different mechanism.

factualpending

Speaker

Andrej Karpathy

Evidence Quote

“I think the paper was even stronger because they hardcoded the weights of a neural network to do gradient descent through attention”

Source

Andrej Karpathy — AGI is still a decade away— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:44 AM

Your notes

Loading notes...