factual
In-context learning resembles gradient descent
In-context learning can be viewed as very similar to gradient descent, with the attention operation acting like gradient descent on the in-context data—n steps of gradient descent match n layers of in-context learning, and loss on tasks like linear regression goes down with shots matched to gradient descent steps.
factualpending
Speaker
Sholto DouglasEvidence Quote
“it looks at in-context learning as basically very similar to gradient descent, but the attention operation can be viewed asgradient descenton the in-context data.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...