causal

Transformer dominance is a self-fulfilling hardware prophecy

Because current GPUs and TPUs are built for very high arithmetic intensity and transformers run well on them, all research concentrates on transformers, creating a self-fulfilling prophecy: a theoretically superior architecture that only achieves half the usable FLOPs on existing chips is worthless even with a 30% compute-efficiency win, because it runs half as fast on the hardware.

causalpending

Speaker

Dylan Patel

Evidence Quote

“if it's a 30% compute efficiency win, it's half as fast on the chip.”

Source

@Asianometry & Dylan Patel — How the semiconductor industry actually works— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:32 AM

Your notes

Loading notes...