factual

LLMs use a crude form of RL without value functions

Current LLM RL is conceptually a very dumb form of RL — every token in a successful trajectory is upweighted — and notably lacks value functions, even though far more sophisticated methods like temporal difference / Q-learning existed a decade ago (e.g. Atari game-playing); it is surprising that this crude approach works as well as it does.

factualpending

Speaker

Adam Marblestone

Evidence Quote

“conceptually it's a really dumb form of RL, even compared to what was being done 10 years ago. Even the Atari game-playing stuff was using Q-learning, which is basically a kind of temporal difference learning.”

Source

Adam Marblestone – AI is missing something fundamental about the brain— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:32:22 AM

Your notes

Loading notes...