factual

RLVR Follows a Scaling Law, RLHF Does Not

RLVR follows a scaling paradigm where log-increasing training compute yields a linear increase in evaluation scores (reproduced by OpenAI's o1 and DeepSeek), but there is no such scaling law for RLHF—in fact the seminal RLHF scaling paper is about reward model over-optimization—which is field-defining because to do the best RLVR you need the extra 10-100x compute but for RLHF you don't.

factualpending

Speaker

Nathan Lambert

Evidence Quote

“the seminal scaling paper for RLHF is about scaling loss for reward model over-optimization. That’s a big line to draw with RLVR”

Source

Ai Sota #2026— Lex Fridman Podcast
Created: 6/13/2026, 3:24:41 AM

Your notes

Loading notes...