transformers
The neural-network architecture, built by stacking layers, that underlies modern large language models and does the internal calculations of the AI.
As defined in Zvi Mowshowitz on AI and the Dial of Progress
Other definitions
Neural network architecture based on attention mechanisms that has become the dominant paradigm for large language models and other deep learning applications.
OpenAI CEO Sam Altman | AI for the Next Era
Where it comes up · 21
- The coming AI revolution and the question no one can ignore | Max Tegmark | Strange Loop4 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough3 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold3 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold3 claims
- Demis Hassabis: Why AGI is Bigger than the Industrial Revolution & Where Are The Bottlenecks in AI3 claims
- Ilya Sutskever: Deep Learning | Lex Fridman Podcast #942 claims
- Applied Intuition: A Billion Intelligent Machines2 claims
- Zvi Mowshowitz on AI and the Dial of Progress2 claims
and 13 more
Most important claims · 10 of 39Search all claims for “transformers” →
- Historical papers from Shane and Hassabis (2009 and 2016) anticipated key developments: Shane's thesis identified compression of Wikipedia as a test for AI (which became the loss function for LLMs), and Hassabis's 2016 neuroscience-AI paper identified attention as a critical capability (which was later crystallized in transformers).Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Roughly 10 years after Hinton's 1985 model, Yoshua Bengio showed the same feature-vector approach worked on real English sentences with 5-10 input words and predicted the next word about as well as the best language models of the time; about 10 years later the linguists conceded feature vectors were a good idea; and about 10 years after that Google invented transformers.Will AI outsmart human intelligence? - with 'Godfather of AI' Geoffrey Hinton
- Cuba's backup power system consists of diesel generators distributed across the island and plugged into the main grid, but these generators do not produce steady current, putting extreme stress on transformers and transmission systems that have also gone 50 years without meaningful maintenance.Cuba's Power Grid Fails, But an Opportunity Arises || Peter Zeihan
- Shane Legg's 2009 thesis proposed that a test for AI is whether you can compress Wikipedia, which is essentially the loss function used to train modern LLMs, and Hassabis's own 2016 paper comparing neuroscience and AI identified attention as a needed capability before transformers existed.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- The reason AI systems are effective is not the algorithms (gradient descent and transformers) but the data: hundreds of thousands of humans who put text and images on the web, plus reinforcement learning from human feedback and prompt engineering, supply the human exploratory and creative intelligence; tech companies then credit the algorithms much like the 'Stone Soup' folktale, in which the magic stone gets credit for soup actually made from villagers' contributions.308 | Alison Gopnik on Children, AI, and Modes of Thinking
- Learning to recognize terminology like supervised/unsupervised learning and transformers helps you work out what the best tool is for the job, preventing wasteful use of overly complex models when simpler approaches suffice.You need to learn AI in 2024! (And here is your roadmap)
- Google's early period of development and careful deployment of Transformers and fusion models (2017 onwards) created a temporary holiday from public release, but this monopoly advantage ended once OpenAI released ChatGPT with Microsoft backing, forcing Google to compete and release similar systemsPossible End of Humanity from AI? Geoffrey Hinton at MIT Technology Review's EmTech Digital
- Transformers, invented by Google Research and Brain colleagues, enabled scaling to massive amounts of data, fundamentally accelerating modern AI progress and representing a key inflection point.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- While a pure-RL, no-priors AlphaZero-style path to AGI is theoretically possible, the quickest and most plausible path is to use all existing world knowledge from the Web via scalable algorithms like transformers as a prior, then add planning and search on top; the final AGI will likely include large multimodal models but not rely on them alone.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- From the start DeepMind bet on generality and learning, triangulating on reinforcement learning, search, and deep learning as algorithms that scale, are general, and don't require handcrafted human priors—deliberately moving away from the 1990s logic-based expert systems whose hand-coded information proved too rigid; AlphaGo demonstrated these systems were ready to scale and transformers later turbocharged progress.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold