normative

Alignment succeeding too well is a risk

There is a danger in alignment succeeding too well: whoever ends up in charge of AI systems—companies or governments—would gain fine-grained control over autonomous minds (e.g. turning up a 'loyalty feature'), raising both the ickiness of controlling an autonomous mind and the value lock-in risk, which is a strong reason to be extremely open and publish the constitutions models are expected to abide by.

normativepending

Speaker

Sholto Douglas

Evidence Quote

“directions likepublishing the constitutionthat you expect your model to abide by–trying to make sure that you RLHF it towards that, and ablate that, and have the ability for everyone to offer feedback and contribution to that–is really important.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...