forecast
Transformer architecture likely to persist
The core self-attention transformer architecture has not fundamentally changed in six or seven years, apart from becoming mixture-of-experts, and fixing its quadratic attention cost would add costs elsewhere, so innovation will likely come from data and auxiliary layers rather than a new transformer.
forecastpending
Speaker
AravindEvidence Quote
“it's very hard to make an innovation on the transformer that can have a material impact at the level of GPT-4”
Source
Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]— Invest Like the BestCreated: 11/14/2026, 5:47:28 AM
Your notes
Loading notes...