transformer architecture
The neural-network architecture introduced in 2017 that underpins modern large language models.
As defined in Demis Hassabis #2
Other definitions
The neural network design underlying modern large language and reasoning models, enabling capabilities like clinical transcription and downstream task automation.
313 | Eric Topol on the Changing Face of Medicine and AgingA neural network architecture based on attention mechanisms that processes sequential data in parallel, forming the foundation of modern large language models like ChatGPT.
Finally an NVIDIA Competitor for AI? Maybe, RAIN's Brain chips!!!
Where it comes up · 15
- Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History12 claims
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development4 claims
- This is not the AI we were promised | The Royal Society2 claims
- Demis Hassabis #22 claims
- 280 | François Chollet on Deep Learning and the Meaning of Intelligence2 claims
- 313 | Eric Topol on the Changing Face of Medicine and Aging2 claims
- Ex-Google CEO: What Artificial Superintelligence Will Actually Look Like w/ Eric Schmidt & Dave B2 claims
- Generative AI in a Nutshell - how to survive and thrive in the age of AI1 claim
and 7 more
Most important claims · 10 of 34Search all claims for “transformer architecture” →
- AI history teaches that it is easy to get over-excited and read too much into breakthroughs; people have repeatedly believed that a single technique (search in the 1950s, deep learning a decade ago, now Transformers) is the 'magic ingredient' for AGI, but each time the field discovers there are missing ingredients we don't yet know about.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- State-of-the-art LLMs are roughly four orders of magnitude less efficient than the human brain in converting energy to computation, which implies a better, far more energy-efficient architecture exists and will likely be found long before AGI.Pat Grady - Relentless Application of Force - [Invest Like the Best, EP.378]
- The Transformer architecture is fundamentally a 'relative machine' that compares words against each other based on context rather than against fixed filters; this allows it to capture linguistic relationships essential to meaning, making it a major conceptual leap beyond previous neural network approaches.AI: Grappling with a New Kind of Intelligence | World Science Festival
- The Transformer architecture was designed for token prediction (predicting the next word in a sequence) and is coupled with enormous compute and data; despite not being designed for logical reasoning, planning, or robotics, Transformers have proven to be a powerful general-purpose tool.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Current AI systems can do incremental hill-climbing along an S-curve but no one has unequivocally shown they can make a big leap like inventing the Transformer architecture; whether scaling existing approaches is sufficient for AGI or one or two more breakthroughs are needed is roughly a 50/50 open question, which is why DeepMind devotes about half its resources to blue-sky ideas and half to scaling current capabilities.Demis Hassabis #2
- The transformer was a natural combination of ideas, such as attention and residual networks, that had independently been effective, so none of its core pieces were surprising to researchers in the field.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- Large language models are disembodied systems: they have no persistent existence in the world, no goals or desires beyond responding to the current input, and no coupling with real-world consequences—making them fundamentally limited compared to embodied intelligent agents.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Before the 2017 transformer paper, attention, residual networks and other ideas were already known, and the transformer was only obviously better once half a dozen additional tweaks were combined, which required research taste and many small experiments.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- A secure transformer architecture that protects against prompt injection attacks slows training by a factor of 2-3x, and homomorphic encryption for truly private AI slows computation by several hundred times, making provable security extremely expensiveFacing Superintelligence (with Ben Goertzel)
- The Transformer architecture was designed specifically for token prediction (next-word prediction in language models) by Google researchers in 2017; its remarkable capabilities emerged when coupled with massive data and compute, but this does not mean Transformers are the key ingredient for other AI tasks like logical reasoning or robotics.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History