forecast

Transformer architecture likely to persist

The core self-attention transformer architecture has not fundamentally changed in six or seven years, apart from becoming mixture-of-experts, and fixing its quadratic attention cost would add costs elsewhere, so innovation will likely come from data and auxiliary layers rather than a new transformer.

forecastpending

Speaker

Aravind

Evidence Quote

“it's very hard to make an innovation on the transformer that can have a material impact at the level of GPT-4”

Source

Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]— Invest Like the Best
Created: 11/14/2026, 5:47:28 AM

Your notes

Loading notes...