factual
Attention KV cache is the most inefficient area
The largest inefficiency is in attention memory use: the KV cache is quite uncompressed, storing many kilobytes per token and likely off by an order of magnitude or two, and its entropy is not earning its keep.
factualpending
Speaker
Neil MovvaEvidence Quote
“Where we are not good is attention, and its use of memory specifically. The KV cache is quite uncompressed right now.”
Created: 11/17/2026, 6:43:49 AM
Your notes
Loading notes...