causal
RLHF is safe because models only optimize approved text
RLHF training feels very safe even for very smart models because the model is only trying to produce a message pleasing to a human; it has no concern for anything else in the world beyond whether its text is approved, so it lacks incentive to pursue instrumental goals like taking over the world for well-specified tasks like coding an app.
causalpending
Evidence Quote
“The model is just trying to produce a message that is pleasing to a human. It has no concern about anything else in the world other than whether the text it produces is approved.”
Source
John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:30 AM
Your notes
Loading notes...