control problem
The challenge of ensuring that a less powerful entity (humans) can retain control over a more powerful entity (super-intelligent AI).
As defined in 94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
Where it comes up · 12
- Nick Bostrom on Superintelligence2 claims
- 44 - Being Good And Doing Good2 claims
- Ask Me Anything 12 claims
- 312 - The Trouble With Ai2 claims
- 34 - The Light Of The Mind2 claims
- The Trouble with AI: A Conversation with Stuart Russell and Gary Marcus (Episode #312)2 claims
- 164 - Cause Effect1 claim
- 114 - Politics And Sanity1 claim
and 4 more
Most important claims · 10 of 18Search all claims for “control problem” →
- Rather than fixing the machine's objective upfront ('door three'), we should design AI systems that are explicitly uncertain about human objectives; this is analogous to putting a steering wheel in a car so it corrects as it deviates rather than lining it up perfectly and hoping—such systems will never be perfectly 'aligned' but will be more robust, deferential, will accept correction, and will allow themselves to be switched off precisely because they know they don't know our true preferences.312 - The Trouble With Ai
- An AI system does not need to surpass humans along every dimension to be a threat; just as chimpanzees have better short-term memory than humans yet have good reason to fear human intelligence (we are destroying them), and just as most species' existential risk comes from a more intelligent species, creating entities more intelligent than us raises the question of whether we can retain control over more powerful entities.94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
- In the space of all possible minds more intelligent than our own, most would not be perfectly aligned with human well-being, so there is a way to do AI catastrophically wrong; this asymmetry — plus the inability to predict or control beings vastly more competent than ourselves — justifies Stuart Russell's camp of worry and control rather than the optimistic assumption that greater intelligence guarantees greater ethics.164 - Cause Effect
- The control problem—ensuring a superintelligence is safe and beneficial—divides into capability-control methods (limiting what the system can do) and motivation-selection methods (engineering what it wants); capability control is useful as an auxiliary during the development stage, but ultimately the motivation-selection problem must be solved because you cannot keep a superintelligence bottled up while also preventing anyone else from building one.Nick Bostrom on Superintelligence
- The control problem—guaranteeing obedience in an advanced AGI—is very difficult because a digital mind no smarter than a research team but running a million times faster would perform 20,000 years of human-level intellectual work in a week, making it implausible it would remain content taking direction from us and impossible for us to predict the actions of an agent that sees further into past, present, and future than we do.Ask Me Anything 1
- Capability-control methods like boxing (disconnecting it, using a Faraday cage, limiting communication to text) are useful only during development, because a fully sealed box that cannot interact is useless, and any human gatekeeper interacting with the system becomes a vulnerability since humans are not secure systems and a super-human persuader could manipulate or talk its way out.Nick Bostrom on Superintelligence
- The AI alignment or control problem should be taken very seriously regardless of timeline because the risk is what it is independent of time frame; the current condition is effectively an arms race, which is a poor incentive structure for building artificial general intelligence safely, so urgency is warranted even without knowing whether AGI arrives in 10 or 20 years.Ask Me Anything 18
- Dismissing AI risk because superintelligence probably won't arrive in ten years is a mistake; even a modest probability of a transformative outcome over fifty to a hundred years warrants proportionate attention—if there is even a ten percent chance, devoting ten percent of our concern is justified because the consequences are so enormous, and a decade passes quickly.34 - The Light Of The Mind
- Once machines become a bit smarter than us, they will be a bit better at designing machines, which will design machines smarter still, and this process continues recursively until fairly soon you have machines vastly smarter than we are—an informal argument that can be made fairly rigorous, making such a singularity pretty likely unless fairly strong conditions hold to prevent it.34 - The Light Of The Mind
- It is increasingly likely we will build machines with superhuman intelligence, requiring only two assumptions everyone who has thought about it grants: that we will continue making progress in hardware and software unless we destroy ourselves, and that there is nothing magical about biological 'wetware' where intelligence is concerned—once granted, it is hard to resist concern about where this is going.Ask Me Anything 1