causal
Minimum viable intelligence is smaller than minimum viable alignment
Channeling Byrnes, the minimum set of Steering Subsystem components needed to produce something intelligent is far smaller than the set needed for human-like social instincts and ethics; therefore it is probably possible to build powerful model-based RL optimizers (potential paperclip maximizers) lacking most human reward functions, which is why understanding the Steering Subsystem matters chiefly for alignment.
causalpending
Speaker
Adam MarblestoneEvidence Quote
“it probably is possible to make super powerful model-based RL optimizing systems and stuff like that that don't have most of what we have in the human brain reward functions and as a consequence might want to maximize paperclips. And that's a concern.”
Source
Adam Marblestone – AI is missing something fundamental about the brain— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:32:22 AM
Your notes
Loading notes...