factual
Context tokens are sharp; parameters are squishy
Information memorized in model parameters is memory-efficient (roughly one fact per parameter) but squishy, leading to hallucination and confused facts because trillions of tokens are churned together; information in the context window is sharp and clear thanks to the attention mechanism, but each token costs far more memory (kilobytes to a megabyte of keys and values per token across layers), creating a fundamental trade-off.
factualpending
Speaker
Jeff DeanEvidence Quote
“it's all a bit squishy because you've churned all these tokens together. The model has a reasonably clear view of that data, but it sometimes gets confused... Whereas information in the context window... is really sharp and clear”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...