causal

RLHF Averages Out Voice and Incisiveness

Language models struggle to be incisive or have a distinctive voice because RLHF is designed to take feedback from many people and average how the model behaves; this creates a fundamental knot in the RLHF problem formulation that provides utility but prevents the model from having a deep, raw expression like a researcher trying to put a feeling into words.

causalpending

Speaker

Nathan Lambert

Evidence Quote

“This is a wonderful fundamental problem for researchers in RLHF: this provides so much utility in making the models better, but also the problem formulation has this knot in it that you can’t get past.”

Source

Ai Sota #2026— Lex Fridman Podcast
Created: 6/13/2026, 3:24:41 AM

Your notes

Loading notes...