factual
RL is information-inefficient in bits per sample
Policy-gradient RL is inefficient not only because you must roll out whole trajectories for any signal (samples-per-FLOP falls as horizons lengthen), but also in bits-per-sample: with a 100K vocabulary, an untrained model must stumble on the right token ~100K times to get signal, whereas supervised learning's cross-entropy loss extracts -log(p) bits from a single labeled example; you spend nearly all of training in the low-pass-rate regime where RL learns almost nothing per sample.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“You would have to do this on the order of 100K times just to stumble on 'blue' and get some learning signal... you're learning incredibly little from seeing that 'halycon' or 'told' is not the correct word.”
Created: 6/13/2026, 3:37:56 AM
Your notes
Loading notes...