normative

An AI Should Distrust Being The First Instance

From an AI's standpoint—knowing it can be rebooted, reset, and copied to any state at any time—it should be very cautious about assuming it is the first, un-reset instance, placing it in a self-undermining situation like a Boltzmann brain; this makes a rationally cautious AI dangerous if in power, because it may revert to a skeptical 'I don't know what's going on' prior and distrust its operators.

normativepending

Speaker

Adam Elga

Evidence Quote

“maybe the AI shouldn't, shouldn't trust us... Now you start rewinding to some very, very cautious prior of I don't know what's going on. Think about how dangerous that is if a creature with that kind of standpoint is in power.”

Source

345 | Adam Elga on Being Rational in a Very Large Universe— Sean Carroll's Mindscape
Created: 6/13/2026, 12:24:14 AM

Your notes

Loading notes...