factual

Transformers were 100x more efficient, enabling GPT

OpenAI's GPT insight was that the transformer made language modeling roughly 100x more efficient at reaching the same performance than RNNs, so running the same compute yields a far better model; combined with scaling laws and the Chinchilla insight that you must scale both parameters and tokens (and token quality/dedup), this evolution — not attention alone — produced the breakthrough.

factualpending

Speaker

Aravind Srinivas

Evidence Quote

“It’s not like a attention alone was important. Attention, parallel computation, transformer, scaling it up to do unsupervised pre-training, right data and then constant improvements.”

Source

Aravind Srinivas— Lex Fridman Podcast
Created: 6/13/2026, 3:26:05 AM

Your notes

Loading notes...