causal
KV Cache Grows Larger Than Model Weights
The KV cache stores a representation of every token in the conversation and often becomes larger than the model weights, and because its size depends on users and context, it is the main limit on purely SRAM-based low-latency serving.
causalpending
Speaker
Neil MovvaEvidence Quote
“KV cache is going to be a big thorn in your side.”
Created: 11/17/2026, 6:43:49 AM
Your notes
Loading notes...