factual

Attention cost dominated by MLP block

The quadratic cost of attention is overblown: in typical dense transformers it is dominated by the MLP block, attention's n-squared term only matters at very long context, and at inference time generation is actually linear because one set of Q-vectors looks up many KV-vectors—so the 'quadratic attention is fatal' meme is misleading.

factualpending

Speaker

Sholto Douglas

Evidence Quote

“the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...