factual
In-context learning may run internal gradient descent
In-context learning is pattern completion within a token window that emerges from pre-training, and there is evidence (e.g. a linear-regression-in-context paper) that it may internally run a small gradient-descent-like loop in the layers of the network rather than being a fundamentally different mechanism.
factualpending
Speaker
Andrej KarpathyEvidence Quote
“I think the paper was even stronger because they hardcoded the weights of a neural network to do gradient descent through attention”
Created: 6/13/2026, 3:26:44 AM
Your notes
Loading notes...