factual

Transformer Was Combination of Known Ideas

The transformer was a natural combination of ideas, such as attention and residual networks, that had independently been effective, so none of its core pieces were surprising to researchers in the field.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“At the time, my sense is that the transformer architecture was a natural combination of many ideas (attention, residual networks) which had independently been effective, taken to quite aggressive lengths.”

Source

Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:27:26 AM

Your notes

Loading notes...