normative
RLHF located weird corporate-speak, not niceness
There is a narrow ray of hope where, instead of having Amazon Turks label outputs (which located only a weird, corporate-speak, left-rationalizing telephone-announcement creature in current RLHF), you train a model to distinguish people being nice, sensible, and arguing validly using data from a dozen carefully chosen people and good fictional characters, filter training data to only the light in humanity, and use bootstrapping reflection so the mask eats the shoggoth, yielding a genuine ally; but this only works if you don't amp it up too far and isn't the easy genie-grants-wishes alignment.
normativepending
Speaker
Eliezer YudkowskyEvidence Quote
“Consciousness is when the mask eats the shoggoth. Maybe that’s it, maybe with the right set of bootstrapping reflection type stuff you can have that happen on purpose more or less, where the system’s output that you’re shaping is to some degree in control of the system and you locate niceness in the human space.”
Source
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:29:05 AM
Your notes
Loading notes...