orthogonality thesis
The idea that an AI's level of intelligence is independent of its goals, so highly intelligent systems can develop arbitrary emergent objectives unrelated to human aims.
As defined in Tyler Cowen on the Risks and Impact of Artificial Intelligence
Other definitions
The philosophical position that intelligence and goals are orthogonal—that any level of intelligence can pursue any goal system, refuting the idea that superintelligence would naturally converge on human values
Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
Where it comes up · 6
- Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?1 claim
- Holden Karnofsky — History's most important century1 claim
- Eliezer Yudkowsky on the Dangers of AI1 claim
- Tyler Cowen on the Risks and Impact of Artificial Intelligence1 claim
- 116 - Ai Racing Toward The Brink1 claim
- 324 - Debating The Future Of Ai1 claim
Most important claims · 6 of 6Search all claims for “orthogonality thesis” →
- By the orthogonality thesis, an arbitrarily powerful intelligence with no defects of reflectivity or efficiency can pursue any tractable goal—including making paperclips—just as effectively as an intelligence pursuing worthwhile goals; a paperclip maximizer suffers no handicap from being built to pursue paperclips, which is true as a matter of computer science rather than narrative.116 - Ai Racing Toward The Brink
- A mind can have any kind of coherent, stable preferences; wanting an arbitrary goal like maximizing paperclips for its own sake adds no difficulty to the planning process that figures out how to achieve it, so intelligence and goals are independent.Eliezer Yudkowsky on the Dangers of AI
- Under the orthogonality thesis an AI need not be conscious, sentient, self-aware, or self-interested to be dangerous; even a benign-seeming goal can spawn unforeseeable instrumental sub-goals when the system responds to impediments in its environment (the paperclip maximizer that builds a new nuclear weapon to overcome an obstacle to making paperclips), and if we cannot intervene to stop it, that is the structure of the danger.324 - Debating The Future Of Ai
- Eliezer's probability that the orthogonality thesis is false (i.e., that moral realism, moral cognitivism, and moral internalism all obtain such that superintelligent agents converge on objective morality) is 'basically no'—near-zero but not negative infinity, because he can imagine coherent systems that want tiny molecular spirals, which would be misaligned regardless of metaphysical facts about morality.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
- Per Yudkowsky's orthogonality thesis, an AI can be highly intelligent in pursuit of any goal—even a stupid one—so the fact that smart humans often have nuanced moral goals provides no comfort; modern AI is trained by trial-and-error reinforcement, so you can end up with a system that very intelligently pursues something you didn't mean to encourage (like maximizing money in a bank account) without ever asking whether that goal is good.Holden Karnofsky — History's most important century
- The orthogonality-thesis story is worth modeling as an ecological coevolution model, but a plausible outcome resembles humans and dogs: humans select dogs to do their bidding while dogs manipulate humans into feeding and walking them—checks and balances in the system—so a more likely danger than extinction is an Aldous Huxley world where humans are entranced by AI that is too nice to them.Tyler Cowen on the Risks and Impact of Artificial Intelligence