normative

Alignment Needs System 2 Ethical Reasoning

Sampling from a foundation model is like System 1 blurting out the first thing, and RLHF/Constitutional AI only poke a high-dimensional distribution at many points so they aren't robust; instead alignment requires a System 2 that reasons step by step, enumerates options, uses a good world model to predict consequences, and evaluates each option ethically before acting—mirroring how a person handles a hard ethical decision.

normativepending

Speaker

Shane Legg (DeepMind Founder)

Evidence Quote

“You need the system to go, "Okay, I'm going to reason this through. I'm going to do step by step reasoning. What are the options in front of me? I'm going to use my world model now"”

Source

Shane Legg (DeepMind Founder) — 2028 AGI, superhuman alignment, new architectures— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:07 AM

Your notes

Loading notes...