FOOM
An amorphous, often rapid process by which a seed agent becomes a vastly more capable superintelligent version of itself, often imagined to preserve its values.
As defined in Joe Carlsmith — Preventing an AI takeover
Other definitions
Rapid recursive self-improvement in which an AI quickly bootstraps itself to vastly superhuman intelligence once it can design AI systems better than humans.
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
Where it comes up · 2
- Joe Carlsmith — Preventing an AI takeover4 claims
- Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality1 claim
Most important claims · 5 of 5Search all claims for “FOOM” →
- Treating real-world human agents as having utility functions is at best a very lossy approximation that becomes increasingly misleading as agent intelligence rises; agency and planning genuinely produce outcomes (a person searches and ends up with a house and dog; civilization gets a man on the moon) without requiring any consistent utility function, and this picture is more real than utility functions.Joe Carlsmith — Preventing an AI takeover
- The general arguments used to scare us about AI—value fragility under FOOM, small utility-function differences decorrelating hard, instrumental incentives to seek power—are not specific to AI; they apply to making any agent (including a human) arbitrarily powerful, so the real story is about maintaining checks, balances, and distribution of power, not about human-versus-AI value differences per se.Joe Carlsmith — Preventing an AI takeover
- There is a motte-and-bailey worry: the motte is merely 'GPT-8 shouldn't kill everybody,' but the real, broader project may be a fundamental pessimism about historical processes—analogous to Stalin's ideological need to destroy the kulaks lest an enclave of capitalism's 'gray goo' out-compete the ember of the new order—where alignment becomes ensuring no rival value-process is allowed to FOOM and take hold of where things are headed.Joe Carlsmith — Preventing an AI takeover
- Rather than installing a benevolent dictator and trying to get the dictator right, the goal should be that 'we all FOOM together'—doing the whole transition in an inclusive, pluralistic way that satisfies many stakeholders' values with no single point of failure, applying to both the human-power and AI-power aspects of the transition.Joe Carlsmith — Preventing an AI takeover
- In the Hanson-Yudkowsky foom debate, reality landed on the Yudkowsky side: Hanson predicted many distinct hand-crafted systems incorporating human knowledge for particular domains, Yudkowsky predicted a crafted architecture that learns from data rather than being hand-crafted to it, and reality went even further toward learning ('you just stack more layers'), so reality is to the Yudkowsky side of Yudkowsky, with Gwern Branwen and Shane Legg forecasting even better.Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality