causal

RLHF is safe because models only optimize approved text

RLHF training feels very safe even for very smart models because the model is only trying to produce a message pleasing to a human; it has no concern for anything else in the world beyond whether its text is approved, so it lacks incentive to pursue instrumental goals like taking over the world for well-specified tasks like coding an app.

causalpending

Speaker

John Schulman (OpenAI Cofounder)

Evidence Quote

“The model is just trying to produce a message that is pleasing to a human. It has no concern about anything else in the world other than whether the text it produces is approved.”

Source

John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:30 AM

Your notes

Loading notes...