factual
Attention cost dominated by MLP block
The quadratic cost of attention is overblown: in typical dense transformers it is dominated by the MLP block, attention's n-squared term only matters at very long context, and at inference time generation is actually linear because one set of Q-vectors looks up many KV-vectors—so the 'quadratic attention is fatal' meme is misleading.
factualpending
Speaker
Sholto DouglasEvidence Quote
“the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...