causal
Transformer needed combining existing ideas and tweaks
Before the 2017 transformer paper, attention, residual networks and other ideas were already known, and the transformer was only obviously better once half a dozen additional tweaks were combined, which required research taste and many small experiments.
causalpending
Speaker
Dwarkesh PatelEvidence Quote
“no one knew how to combine the puzzle pieces. Doing so required research taste, of course, but also tons and tons of small experiments.”
Source
Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:27:26 AM
Your notes
Loading notes...