Transformer
A neural network architecture, introduced by Google researchers in 2018, that is easy to train in parallel on large GPU clusters and underpins modern large language models.
As defined in et_995
Other definitions
A neural network architecture, invented in 2017, that processes sequences in a generalized way and underlies modern large language models like GPT.
Marc Andreessen on Why AI Will Save the World
Where it comes up · 53
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT12 claims
- Illustrated Guide to Transformers Neural Network: A step by step explanation10 claims
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]7 claims
- What Is ChatGPT Doing? Episode 1: Introduction6 claims
- How LLMs Took Over The World6 claims
- 248 | Yejin Choi on AI and Common Sense5 claims
- The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap5 claims
- Ilya Sutskever: Deep Learning | Lex Fridman Podcast #944 claims
and 45 more
Most important claims · 10 of 129Search all claims for “Transformer” →
- The transformer, the core algorithm of large language models, was invented at Google and published in 2017, but Google did not productize it citing safety, so OpenAI picked it up and ran with it, and Anthropic and other US labs also build on work they did not create.Marc Andreessen - The Battle For Tech Supremacy - [Invest Like the Best, EP.410]
- In a transformer, input words are turned into high-dimensional vectors (embeddings) that flow through the network; the attention mechanism repeatedly massages these vectors layer by layer so each word's vector incorporates contextual information from the others, so that by the final layer the last vector carries enough context to predict the next word correctly (e.g., 'homework' after 'the dog ate my').336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- Google's 2017 'transformer' architecture distills intelligence from massive data, turning AI into heavy industry requiring giant data centers full of NVIDIA equipment running 24/7—like an oil refinery—which multiplied demand for NVIDIA chips by 100x to 1000x and rocketed its valuation from billions to trillions.The Man Who Built NVIDIA (with Stephen Witt)
- Transformers addressed the fundamental limitations of recurrent neural networks in learning long-term dependencies, and this technical breakthrough immediately made clear that GPT development should switch to transformers as the core architecture.The Mastermind Behind GPT-4 and the Future of AI | Ilya Sutskever
- Pretraining on next-word prediction across trillions of tokens forces a model to store many concepts, and a second stage of RLHF on helpful, honest examples turns that base into an assistant, with the second stage using far less data and compute.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- It is not a coincidence that data, transformers, and GPUs arrived together: shrinking transistors made personal computers economical, which created the internet and thus mass data, while the same transistor progress drove GPUs through gaming before Nvidia repurposed them for general computation — all dimensions of progress are intertwined and you don't get to pick which improve.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- The transformer simplified neural network design into a homogeneous stack of repeated attention and matrix-multiply layers, so only a few hyperparameters need deciding, which lowered the barrier to entry and shifted the main work from architecture design to data quality.Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]
- Whether a scientific idea proves fruitful cannot be assessed in isolation because it depends on the future and on cultural/societal adoption; like base-ten numerals or the transformer architecture, the winning standard is often not objectively special but becomes entrenched through inertia and standardization, so fruitfulness may never be something you can reinforcement-learn the way you can for localized problems.Dwarkesh Patel Podcast - Terence Tao – Kepler, Newton, and the true nature of mathematical discovery
- Pretraining a model to predict the next word across trillions of examples produces a model that is only moderately accurate (around 18 percent) but must store a large number of concepts internally, which is why it can predict next words in science papers or web text.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- The Transformer architecture is applicable to both language translation (the original use case) and text generation/continuation (as used in ChatGPT), because both involve processing input sequences and generating output sequencesWhat Is ChatGPT Doing? Episode 1: Introduction