factual

Attention KV cache is the most inefficient area

The largest inefficiency is in attention memory use: the KV cache is quite uncompressed, storing many kilobytes per token and likely off by an order of magnitude or two, and its entropy is not earning its keep.

factualpending

Speaker

Neil Movva

Evidence Quote

“Where we are not good is attention, and its use of memory specifically. The KV cache is quite uncompressed right now.”

Source

Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]— Invest Like the Best
Created: 11/17/2026, 6:43:49 AM

Your notes

Loading notes...