causal

Minimum viable intelligence is smaller than minimum viable alignment

Channeling Byrnes, the minimum set of Steering Subsystem components needed to produce something intelligent is far smaller than the set needed for human-like social instincts and ethics; therefore it is probably possible to build powerful model-based RL optimizers (potential paperclip maximizers) lacking most human reward functions, which is why understanding the Steering Subsystem matters chiefly for alignment.

causalpending

Speaker

Adam Marblestone

Evidence Quote

“it probably is possible to make super powerful model-based RL optimizing systems and stuff like that that don't have most of what we have in the human brain reward functions and as a consequence might want to maximize paperclips. And that's a concern.”

Source

Adam Marblestone – AI is missing something fundamental about the brain— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:32:22 AM

Your notes

Loading notes...