factual
Transformer Was Combination of Known Ideas
The transformer was a natural combination of ideas, such as attention and residual networks, that had independently been effective, so none of its core pieces were surprising to researchers in the field.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“At the time, my sense is that the transformer architecture was a natural combination of many ideas (attention, residual networks) which had independently been effective, taken to quite aggressive lengths.”
Source
Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:27:26 AM
Your notes
Loading notes...