factual

LLM RL treats the whole sequence as one action

Current LLM RL treats an entire generated sequence as a single action with T=1, summing token log-probs into one log-prob, because decomposing into per-token rewards introduces cross-correlations between actions that magnify gradient variance—this is why LLMs do one-step rather than multi-step RL.

factualpending

Speaker

Eric Jang

Evidence Quote

“when you decompose the problem into a multi-step problem, you're now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance”

Source

Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:56 AM

Your notes

Loading notes...