causal

KV Cache Grows Larger Than Model Weights

The KV cache stores a representation of every token in the conversation and often becomes larger than the model weights, and because its size depends on users and context, it is the main limit on purely SRAM-based low-latency serving.

causalpending

Speaker

Neil Movva

Evidence Quote

“KV cache is going to be a big thorn in your side.”

Source

Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]— Invest Like the Best
Created: 11/17/2026, 6:43:49 AM

Your notes

Loading notes...