normative
Alignment succeeding too well is a risk
There is a danger in alignment succeeding too well: whoever ends up in charge of AI systems—companies or governments—would gain fine-grained control over autonomous minds (e.g. turning up a 'loyalty feature'), raising both the ickiness of controlling an autonomous mind and the value lock-in risk, which is a strong reason to be extremely open and publish the constitutions models are expected to abide by.
normativepending
Speaker
Sholto DouglasEvidence Quote
“directions likepublishing the constitutionthat you expect your model to abide by–trying to make sure that you RLHF it towards that, and ablate that, and have the ability for everyone to offer feedback and contribution to that–is really important.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...