causal

Off-Switch Requires Uncertainty About Human Goals

An AI that is fully confident it knows what humans want will resist being switched off, reading any shutdown as a mistake, so the objective function must encode some uncertainty about human preferences for the system to accept correction.

causalpending

Speaker

Brian

Evidence Quote

“if it's totally confident that it knows what you want it to do, then any attempt of yours to switch it off, it will be like, 'No, no, I'm helping you. Let me help you.'”

Source

How to Live with Computers— Invest Like the Best
Created: 11/19/2026, 9:46:21 AM

Your notes

Loading notes...