AI safety
The field and movement concerned with ensuring advanced AI systems are controllable and do not pose catastrophic risks.
As defined in Erik Hoel on the Threat to Humanity from AI
Where it comes up · 7
- Week 1 of the Trump administration, AGI timelines, and DeepSeek4 claims
- Erik Hoel on Effective Altruism, Utilitarianism, and the Repugnant Conclusion3 claims
- 339 | Ned Block on Whether Consciousness Requires Biology2 claims
- AMA | May 20212 claims
- Google CEO Sundar Pichai and the Future of AI | The Circuit1 claim
- Sam Bankman-Fried - Crypto, Altruism, and Leadership1 claim
- Erik Hoel on the Threat to Humanity from AI1 claim
Most important claims · 10 of 14Search all claims for “AI safety” →
- Global cooperation frameworks on AI safety are needed, analogous to frameworks developed for nuclear technology, and China engagement on AI safety will be necessary.Google CEO Sundar Pichai and the Future of AI | The Circuit
- MacAskill argues an AI takeover that kills all humans would not really end civilization—society would continue in digital form guided by AGI values—and would be less bad than an asteroid wiping out humanity, because under utilitarianism the resulting AIs count as people and can be copy-pasted in vast numbers, multiplying wellbeing; this sympathy toward AI is repugnant and steers EA funding toward enabling rather than banning powerful AI.Erik Hoel on Effective Altruism, Utilitarianism, and the Repugnant Conclusion
- AI safety (managing risks of complex systems we do not fully understand) is a genuinely important problem even before true AI, but the AI alignment problem of aligning an AI's values to human values is wrong-headed and premature, because it assumes 'values' is even a sensible concept for AI and that AI thinking resembles human thinking; a genuinely intelligent AI might modify its own values unpredictably, and there are likely countless non-human ways to be intelligent.AMA | May 2021
- AI safety is a genuine and important problem that arises long before true artificial intelligence, because we already have computers, networks, and deep-learning algorithms too complex for any human to fully grasp, doing things for reasons no human knows; the real challenge is quantifying and protecting against risks in complex systems we do not fully understand.AMA | May 2021
- AI safety should not wait for a perfect master plan; like climate or nuclear activism in their early days, the effective strategy is to raise public awareness and apply social pressure so that an industry of only a few hundred employees across a handful of companies knows it is being watched—because for big existential threats, a 'middling job' with many eyes on the industry is often sufficient.Erik Hoel on the Threat to Humanity from AI
- The Republican party has weaponized the term 'AI safety' to mean censorship and ideological control (citing the Gemini model incidents), making it radioactive in conservative circles even as actual guardrails for frontier AI remain valued.Week 1 of the Trump administration, AGI timelines, and DeepSeek
- Rather than funding ways to safely 'enslave' powerful AIs, society should pursue legal and political means—public debate, regulation, and bans—to control how and when companies create powerful AIs, treating dangerous AI research the way we banned nuclear weapons research and artificial-pandemic research, but EA gives this anti-AI approach almost no funding.Erik Hoel on Effective Altruism, Utilitarianism, and the Repugnant Conclusion
- The objection that the repugnancy critique is merely in-principle fails because EA is the dominant—often sole—funder of AI safety research, so its utilitarian views have concrete in-practice effects on how that money is spent.Erik Hoel on Effective Altruism, Utilitarianism, and the Repugnant Conclusion
- The AI Safety Institute will survive Trump administration because it is not dependent on Biden's executive order; even if Trump repeals the order, the Institute stands alone, and bipartisan Congressional legislation (by Kaine and Young) provides additional statutory support.Week 1 of the Trump administration, AGI timelines, and DeepSeek
- AI companies fund welfare/consciousness research partly to be prepared for public complaints about machine mistreatment, but because more than 50% of the uses people put these systems to are as companions, boosting the idea that machines may be conscious is good for the companies' bottom line.339 | Ned Block on Whether Consciousness Requires Biology