KV cache
The stored key and value tensors from previous tokens that a transformer reuses during generation to avoid recomputation; its size grows with context length and is a major memory cost.
As defined in Ai Sota #2026
Where it comes up · 15
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served11 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]6 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job4 claims
- What does the next training paradigm look like?2 claims
- Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute2 claims
- Sholto Douglas & Trenton Bricken — How LLMs actually think2 claims
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]2 claims
- Ai Sota #20262 claims
and 7 more
Most important claims · 10 of 39Search all claims for “KV cache” →
- Pipelining cannot reduce per-GPU KV cache memory: while fewer layers per rack means storing fewer layers of KVs, keeping all racks busy requires proportionally more sequences in flight, so the number of micro-batches equals the number of pipeline stages and the pipeline-stage factor exactly cancels—the weight footprint shrinks with pipelining but the activation/KV footprint stays constant.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The first token is slow because the model must reprocess the entire prompt, and for multi-turn chats the whole message history is recomputed, since storing the key-value state in RAM between turns is more expensive than recomputing it.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Long context is fundamentally a no-free-lunch trade-off: an RNN/Mamba single fixed state is cheap but forgets as context grows because it can't compress everything, while transformers remember every token (great for lookup) but are expensive due to the quadratically-growing KV cache and dot product—so progress comes from finding the right Goldilocks ratio of attention layers to compressed states.Ai Sota #2026
- KV cache can be regenerated from scratch (rematerialization, costing a forward pass) or stored across memory tiers (HBM, DDR, flash, spinning disk); the optimal tier balances retrieval cost against holding cost, with the rule that you pick a memory whose drain time (capacity divided by bandwidth) matches how long you intend to hold the data—so cache pricing tiers (e.g., 5-minute vs 1-hour) likely correspond to specific physical memory tiers.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The KV cache stores a representation of every token in the conversation and often becomes larger than the model weights, and because its size depends on users and context, it is the main limit on purely SRAM-based low-latency serving.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Chip design revolves around a memory hierarchy trade-off: off-chip memory gives capacity while on-chip memory gives speed, so hybridizing the two is necessary for models that need both large weights and dynamic context.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- There will be a separation of training and inference compute as the field stabilizes; NVIDIA's Vera Rubin is designed for this with a new chip with very little high-bandwidth memory optimized for pre-fill (the matrix-multiplication-heavy part of inference), lowering cost of ownership per FLOP, while memory is only needed during autoregressive generation with KV cache swaps.Ai Sota #2026
- Batching many users together in a single forward pass is the dominant lever on inference cost: if you do not batch, the per-token economics can be roughly a thousand times worse than if you batch many users together, because the cost of fetching weights from memory is amortized across all batch elements.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Human learning compresses observations into intuitions and big-picture knowledge in the weights, which aids generalization and grokking, whereas perfect-fidelity recall of raw information, as in some savants, can cripple the ability to understand abstractions and metaphors.Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job
- Wafer-scale SRAM architectures can host a trillion-parameter model's weights and reach about 1,000 tokens per second because weights stream in about a millisecond, but the growing, dynamic KV cache is a persistent limit those chips handle poorly.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]