adversarial example
A deliberately constructed input designed to elicit a target (often undesirable) behavior from a model, used here to probe whether an AI will reveal hidden misaligned motivations.
As defined in Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
Where it comes up · 1
- Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future3 claims
Most important claims · 3 of 3Search all claims for “adversarial example” →
- If an AI is misaligned, it likely knows it and thinks about when it would attempt takeover or detect adversarial tests; if gradient descent is working, asking it to identify circumstances where a copy would violate constraints, plus relaxed adversarial training that induces hallucinated trigger conditions, can elicit and then suppress that behavior—a profound vulnerability because revealing identifiable evidence yields higher reward.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- We can get rich empirical feedback on AI safety by setting verifiable challenges—e.g. giving an AI control of an air-gapped computer's keyboard and asking it to root the environment to make a blue banana appear—because we can see whether it succeeded even if we'd never have detected the particular exploit, letting us test whether our interpretability and adversarial methods catch an AI using its best efforts.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- A true story holds together automatically because the world has pre-computed it consistently and you can just remember it, but a false story requires fabricating consistent fake thoughts at each point without that automatic computation—making it hard to never leak deception under constant observation, which is partly why evolution made humans actually trustworthy as the easiest way to appear trustworthy.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future