factual
KV Cache Grows Far Faster Than Pretraining Memory
For Llama 3 70B, the KV cache grows about 320 KB per token of in-context learning, whereas pretraining stores about 0.075 bits per token, a roughly 35-million-fold difference in information stored per token between in-context learning and pretraining.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“So between in-context learning and pre-training there's a 35 million fold difference in the amount of information you're storing per token.”
Source
Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:25:50 AM
Your notes
Loading notes...