attention
A mechanism letting models focus on the most relevant parts of input, which became central to transformers.
As defined in Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
Other definitions
A mechanism in large language models that tells the model which prior tokens to weight most heavily when predicting the next token, enabling context disambiguation.
The Strange Math That Predicts (Almost) Anything
Where it comes up · 10
- The Divided Brain3 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]3 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold2 claims
- Geoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 20242 claims
- What does the next training paradigm look like?1 claim
- What Is ChatGPT Doing? Episode 1: Introduction1 claim
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]1 claim
- The Strange Math That Predicts (Almost) Anything1 claim
and 2 more
Most important claims · 10 of 16Search all claims for “attention” →
- Symbolic AI researchers rejected Hinton's predictive word-vector approach as a 'silly way' to solve language problems, arguing that searching discrete symbol-manipulation rules was superior, but as the scale of problems increased, the feature-interaction approach proved vastly more scalable and effective than symbolic rule-based methodsGeoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 2024
- Shane Legg's 2009 thesis proposed that a test for AI is whether you can compress Wikipedia, which is essentially the loss function used to train modern LLMs, and Hassabis's own 2016 paper comparing neuroscience and AI identified attention as a needed capability before transformers existed.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Modern large language models operate on tokens (letters, words, or punctuation) rather than just an alphabet, predicting the odds of the next token given a string of prior tokens—but unlike simple Markov chains they use 'attention' to decide which prior tokens matter, allowing context like 'blood' and 'mitochondria' to disambiguate that 'cell' means biology rather than a prison.The Strange Math That Predicts (Almost) Anything
- Neuroscience has provided directional inspiration for major AI advances such as reinforcement learning combined with deep learning, experience replay, and attention—not as one-to-one algorithmic mappings but as architectural and representational inspiration; the brain is an existence proof that general intelligence is possible.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Transformers scale because attention lets any token relate to any other, making them general learners that find whatever patterns exist in a sequence, which explains their dominance in language and their ability to scale from millions to trillions of parameters.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Inductive logic programming approaches to learning from symbol strings were less computationally efficient than neural approaches even in the small-scale problems of the 1980s-1990s, but symbolic approaches could not scale to modern problems like ImageNet classification.Geoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 2024
- The left hemisphere uses narrow, sharply focused attention to extract detail without context; it sees objects in relation to their usefulness; it cannot make human connections, understand relationships, humor, tone of voice, or perceive things as unique and individual, instead organizing them into abstract categories and linear connections.The Divided Brain
- The Transformer architecture was proposed in a 2017 paper from Google that introduced the key concept of attention and showed how to apply this architecture to language translation tasksWhat Is ChatGPT Doing? Episode 1: Introduction
- The original design flaw of transformers is placing a fundamentally memory-bound attention layer directly next to a compute-bound MLP layer, and no single chip is good at both, which forces hardware trade-offs.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Architectural innovations like sparse attention and KV cache compaction could provide intermediate representations to solve the memory-scaling problem, and a new architectural innovation is proposed almost every week in recent research.What does the next training paradigm look like?