definition

Weights are hazy recollection, context is working memory

Knowledge stored in the weights is only a 'hazy recollection' of training data due to massive compression (15 trillion tokens into a few billion parameters), whereas tokens in the context window are directly accessible like working memory—which is why feeding a model a full chapter yields far better answers than relying on its trained memory.

definitionpending

Speaker

Andrej Karpathy

Evidence Quote

“anything that you give it as a context at test time is directly in the working memory. That's a very powerful analogy to think through things.”

Source

Andrej Karpathy — AGI is still a decade away— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:44 AM

Your notes

Loading notes...