factual

Naive attention is quadratic and blocks trillion-token context

The naive attention algorithm is quadratic, so it can barely be made to work for millions of tokens even on substantial hardware and has no hope of naively scaling to trillions of tokens; reaching trillion-token context (the entire internet, all a user's personal data, all of Google's source code) will require interesting algorithmic approximations that let the model conceptually attend to vastly more tokens.

factualpending

Speaker

Jeff Dean

Evidence Quote

“the naive attention algorithm is quadratic. You can barely make it work on a fair bit of hardware for millions of tokens, but there's no hope of making that just naively go to trillions of tokens.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...