factual
Transformers were 100x more efficient, enabling GPT
OpenAI's GPT insight was that the transformer made language modeling roughly 100x more efficient at reaching the same performance than RNNs, so running the same compute yields a far better model; combined with scaling laws and the Chinchilla insight that you must scale both parameters and tokens (and token quality/dedup), this evolution — not attention alone — produced the breakthrough.
factualpending
Speaker
Aravind SrinivasEvidence Quote
“It’s not like a attention alone was important. Attention, parallel computation, transformer, scaling it up to do unsupervised pre-training, right data and then constant improvements.”
Created: 6/13/2026, 3:26:05 AM
Your notes
Loading notes...