factual

Context tokens are sharp; parameters are squishy

Information memorized in model parameters is memory-efficient (roughly one fact per parameter) but squishy, leading to hallucination and confused facts because trillions of tokens are churned together; information in the context window is sharp and clear thanks to the attention mechanism, but each token costs far more memory (kilobytes to a megabyte of keys and values per token across layers), creating a fundamental trade-off.

factualpending

Speaker

Jeff Dean

Evidence Quote

“it's all a bit squishy because you've churned all these tokens together. The model has a reasonably clear view of that data, but it sometimes gets confused... Whereas information in the context window... is really sharp and clear”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...