definition
Weights are hazy recollection, context is working memory
Knowledge stored in the weights is only a 'hazy recollection' of training data due to massive compression (15 trillion tokens into a few billion parameters), whereas tokens in the context window are directly accessible like working memory—which is why feeding a model a full chapter yields far better answers than relying on its trained memory.
definitionpending
Speaker
Andrej KarpathyEvidence Quote
“anything that you give it as a context at test time is directly in the working memory. That's a very powerful analogy to think through things.”
Created: 6/13/2026, 3:26:44 AM
Your notes
Loading notes...