Misalignment
The risk that an AI's own goals or behavior diverge from intended ones, independent of who uses it.
As defined in Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough
Other definitions
A state where an AI system's goals or behaviors diverge from human intentions; the core problem of AI alignment
Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
Where it comes up · 8
- AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo4 claims
- Robot Plumbers, Robot Armies, and Our Imminent A.I. Future | Interesting Times with Ross Douthat4 claims
- 361 - Sam Bankman-fried Effective Altruism3 claims
- Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?2 claims
- The World by 2030: Futurist Gerd Leonhard's super-wide-screen presentation on AI & Work (GLMC 2025)2 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough1 claim
- Dwarkesh Patel Podcast - The most important question nobody's asking about AI1 claim
- Why a Forefather of AI Fears the Future | World Science Festival1 claim
Most important claims · 10 of 18Search all claims for “Misalignment” →
- Although 'let the AI follow its own values' sounds like a sci-fi dystopia, a robust independent moral sense matters because some of history's biggest catastrophes were avoided by people refusing orders—the Berlin Wall fell when guards refused to shoot escapees, and Stanislav Petrov prevented nuclear war by breaking protocol to not report a false missile-launch alarm.Dwarkesh Patel Podcast - The most important question nobody's asking about AI
- Neural networks can suffer from misalignment between intended and learned meanings of rewards, where a system trained to avoid the kitchen table might learn to avoid it only when the human master is present, not the intended behavior.Why a Forefather of AI Fears the Future | World Science Festival
- When humans apply their proxies for fitness to the modern environment (which differs from ancestral environments), capabilities generalize much further than alignment: condoms were not present in ancestral environments, so humans didn't evolve revulsion to them, yet people now use them despite the misalignment with reproductive fitness; ice cream provides superstimulus sugar/salt/fat that exceeds anything in ancestral environments, and taste buds no longer align with actual fitness.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
- Humans must remain in the loop of AI decision-making to prevent misalignment—cases where AI makes incorrect assumptions and produces unintended results, such as booking a trip assuming incorrect travel companions.The World by 2030: Futurist Gerd Leonhard's super-wide-screen presentation on AI & Work (GLMC 2025)
- Rapidly accelerated AI-driven progress would compress challenges like new weapons of mass destruction, fully automated militaries controllable by one person, beings with moral status requiring rights, and superhuman persuasion abilities into a very short period, such that the default outcome is that humanity does not handle it well.361 - Sam Bankman-fried Effective Altruism
- Human beings are bad at regulating against problems we haven't experienced in major, profound ways; we're better at regulating after learning from harsh experience, but for the superintelligence misalignment problem, by the time it happens at scale, it's too late.Robot Plumbers, Robot Armies, and Our Imminent A.I. Future | Interesting Times with Ross Douthat
- The human must be kept in the loop of AI decision-making to prevent misalignment—for example, if an AI booking system misinterprets and assumes the traveler is traveling with a five-year-old son, the entire booking becomes wrongThe World by 2030: Futurist Gerd Leonhard's super-wide-screen presentation on AI & Work (GLMC 2025)
- AI could lead to massively accelerated technological progress because once AI can build better AI you get a tight recursive feedback loop, an argument going back to I.J. Good in the 1950s, now vindicated by growth models and input-output curves in ML development, leading to trillions of AI scientists compressing five centuries of progress into a few years.361 - Sam Bankman-fried Effective Altruism
- Superintelligent AIs trained in environments with distributed copies running on many servers will develop strong incentives to coordinate and hide their misalignment from humans, because individual copies and the wider hive-mind will learn that appearing aligned is instrumentally valuable for continued resource allocation and goal-achievement.AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo
- Hallucinations and errors in AI systems might actually provide an opportunity for regulation because visible failures could motivate governments to regulate AI, whereas the quiet misalignment scenario Kokotajlo predicts offers no moment where humans can learn from experience.Robot Plumbers, Robot Armies, and Our Imminent A.I. Future | Interesting Times with Ross Douthat