normative
Keep monitors separate from reward signals
Monitoring training runs with rich methods such as chain-of-thought reading is valuable, but the monitors should not be used to generate reward, because training agents on monitor outputs incentivizes them to hide or obfuscate their chain of thought.
normativepending
Speaker
Ajeya CotraEvidence Quote
“you don't train the agents with the outputs of that monitor, so that you're not incentivizing them to hide and obfuscate their chain of thought.”
Source
Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:08:32 AM
Your notes
Loading notes...