causal

Transformer needed combining existing ideas and tweaks

Before the 2017 transformer paper, attention, residual networks and other ideas were already known, and the transformer was only obviously better once half a dozen additional tweaks were combined, which required research taste and many small experiments.

causalpending

Speaker

Dwarkesh Patel

Evidence Quote

“no one knew how to combine the puzzle pieces. Doing so required research taste, of course, but also tons and tons of small experiments.”

Source

Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:27:26 AM

Your notes

Loading notes...