factual
Naive attention is quadratic and blocks trillion-token context
The naive attention algorithm is quadratic, so it can barely be made to work for millions of tokens even on substantial hardware and has no hope of naively scaling to trillions of tokens; reaching trillion-token context (the entire internet, all a user's personal data, all of Google's source code) will require interesting algorithmic approximations that let the model conceptually attend to vastly more tokens.
factualpending
Speaker
Jeff DeanEvidence Quote
“the naive attention algorithm is quadratic. You can barely make it work on a fair bit of hardware for millions of tokens, but there's no hope of making that just naively go to trillions of tokens.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...