factual

In-context learning resembles gradient descent

In-context learning can be viewed as very similar to gradient descent, with the attention operation acting like gradient descent on the in-context data—n steps of gradient descent match n layers of in-context learning, and loss on tasks like linear regression goes down with shots matched to gradient descent steps.

factualpending

Speaker

Sholto Douglas

Evidence Quote

“it looks at in-context learning as basically very similar to gradient descent, but the attention operation can be viewed asgradient descenton the in-context data.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...