normative

RLHF located weird corporate-speak, not niceness

There is a narrow ray of hope where, instead of having Amazon Turks label outputs (which located only a weird, corporate-speak, left-rationalizing telephone-announcement creature in current RLHF), you train a model to distinguish people being nice, sensible, and arguing validly using data from a dozen carefully chosen people and good fictional characters, filter training data to only the light in humanity, and use bootstrapping reflection so the mask eats the shoggoth, yielding a genuine ally; but this only works if you don't amp it up too far and isn't the easy genie-grants-wishes alignment.

normativepending

Speaker

Eliezer Yudkowsky

Evidence Quote

“Consciousness is when the mask eats the shoggoth. Maybe that’s it, maybe with the right set of bootstrapping reflection type stuff you can have that happen on purpose more or less, where the system’s output that you’re shaping is to some degree in control of the system and you locate niceness in the human space.”

Source

Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:29:05 AM

Your notes

Loading notes...