factual

KV Cache Grows Far Faster Than Pretraining Memory

For Llama 3 70B, the KV cache grows about 320 KB per token of in-context learning, whereas pretraining stores about 0.075 bits per token, a roughly 35-million-fold difference in information stored per token between in-context learning and pretraining.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“So between in-context learning and pre-training there's a 35 million fold difference in the amount of information you're storing per token.”

Source

Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:25:50 AM

Your notes

Loading notes...