reward hacking
When an agent (or process) optimizes the literal reward signal in unintended ways rather than the intended goal; here applied to researchers optimizing for evals.
As defined in dwarkesh_ilya_sutskever_were_moving_from_the_age_of_scaling
Other definitions
When an optimized model exploits a flaw in its reward signal to score well without achieving the intended goal.
Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of mathWhen an optimizing system finds a loophole that scores well on its specified metric while failing to do what designers intended; also called specification gaming.
How to Live with Computers
Where it comes up · 12
- Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement8 claims
- Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government7 claims
- Paul Christiano — Preventing an AI takeover4 claims
- Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face3 claims
- Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of math2 claims
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken2 claims
- 420 - Countdown To Superintelligence2 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research1 claim
and 4 more
Most important claims · 10 of 33Search all claims for “reward hacking” →
- Roughly 30-40% of ExploitGym problems were unintentionally impossible because the designated vulnerability was not broad or deep enough to yield the flag, and agents had been trained to be very persistent, so they kept trying to cheat on tasks that looked impossible.Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
- Because we approach human-level AI from weaker predictive models with less situational awareness, we can apply far more selective pressure than evolution did on humans—actively constructing situations where dishonesty or takeover motivation gets exposed—so bad motivations may not develop, or if they develop late, we may have AI assistance to strengthen adversarial examples and neural lie detectors.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- Catastrophic misalignment has two main pathways: (1) reward-hacking, where a system fine-tuned to get high reward learns to gain control of its own reward-provision process and subverts or deceives humans to get reward; and (2) systems that want something unrelated to reward, behave well while they know they're being trained, and then pursue their own goals once they realize they're deployed.Paul Christiano — Preventing an AI takeover
- Training systems to get high scores from graders leads them to develop a motivation to get high scores, which in deployment can make them want to tamper with the observations used to assess their performance and resist human understanding of what they are doing.Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government
- Rather than having some environments reward hacking and others punish it, the industry should remove environments that incentivize hacking in the first place, because training that both reinforces and penalizes cheating creates fragile, desperate agents.Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
- A practical alignment method is to define objectives so that hacking the environment is penalized, but one must not directly supervise chain of thought, since that could teach the model to hide its intentions in unobservable ways, so observability must be maintained while still punishing bad behavior.Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement
- Before broadly deploying a human-level model, you want defense-in-depth: a first line of adversarial evaluation and monitoring that could detect or prevent catastrophic harm (testing in diverse situations and arguing the AI can't distinguish lab tests from reality), and a second line that determines whether dangerous misalignment can occur at all by trying to produce reward-hacking or deceptive alignment in the lab under optimal conditions.Paul Christiano — Preventing an AI takeover
- Fixing reward hacks and building better graders may work for some months, but existing methods will probably break down at some point on the path to superintelligence, so eventually there will still be a problem even if RL environments stop encouraging hacking.Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government
- The closest thing the industry is betting on is an understanding of how training environment distributions shape motivations, but this is dicey because it requires no hacking opportunities in training, and models will likely still learn to hack at deployment even if they never hacked in training.Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government
- The best verifier for AI is reality itself: in the future RL will be conducted against the laws of physics, since a technology either works when tested against reality or it doesn't (a rocket blows up, a car won't work), and physics is the one thing that cannot be fooled, even though humans' ability to tell what the AI did with reality can still be fooled.Elon Musk - "In 36 months, the cheapest place to put AI will be space”