Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
What this covers
Eric Jang walks through how AlphaGo works from first principles, treating it as a fully reproducible system he rebuilt on modest compute. The conversation moves from the core mechanics of the algorithm—how MCTS provides stable training targets and how value functions compress expensive search into a single forward pass—outward to what AlphaGo reveals about intelligence itself. The argument is that Go's discrete action space and concretely groundable outcomes made it uniquely suited to this approach, and that understanding why creates a lens for seeing what current automated AI research systems can and cannot do.
The substance spans neural-network amortization of hard search problems, the mechanics of distillation and self-play refinement, and the role of off-policy training in stable supervised learning. Jang examines where AlphaGo's design choices matter and where they don't—including which tricks from strong systems like KataGo are architectural and which are merely transitory wins on specific hardware. He contrasts this with continuous-control and language-model settings, where the credit assignment problem and the sheer inefficiency of bits per sample in policy-gradient RL make MCTS as policy-improvement operator infeasible. The discussion turns contentious around whether compute multipliers stack predictably, whether the Bitter Lesson applies equally everywhere, and how much human guidance frontier coding agents still require to steer research direction rather than merely execute specified experiments.
Eric Jang argues that AlphaGo remains the cleanest worked example of intelligence primitives because MCTS provides a strictly-better supervised target for every move—sidestepping the credit-assignment and exploration problems that make LLM policy-gradient RL information-inefficient—and that this contrast illuminates what current automated AI research can and cannot yet do.
- MCTS relabels every action with a strictly better target, turning RL into stable supervised learning rather than 'sucking supervision through a straw'
- MCTS works for Go because value is concretely groundable and the discrete action space suits PUCT exploration, conditions absent in LLM/continuous-control settings
- Automated coding agents already excel at running experiments and hyperparameter search but fail at choosing the next question and escaping research dead ends
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The profound accomplishment of AlphaGo is that roughly ten sequential layers of neural-network computation can amortize and approximate to high fidelity a nearly intractable, NP-class search problem—suggesting our worst-case-complexity framing of hard problems may be incomplete, since hard problems often have exploitable structure.
“10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem.”
Even though Go and weather are chaotic and highly sensitive to initial conditions (a single stone or perturbation changes the exact future), the macroscopic quantity we care about—who wins, where the hurricane goes—is predictable, like the Lorenz attractor whose shape is knowable even when the exact trajectory is not; this is what makes value functions learnable in such systems, unlike a hash function which has no macrostructure.
“If you start anywhere on the Lorenz attractor, you don't know where you're going to end up, but you do know that the thing looks like this.”
MCTS acts as a policy-improvement operator: for every action taken in a game (whether the game was won or lost), the search provides a strictly better action/distribution to imitate, giving one low-variance supervision target per move rather than a single noisy win/loss signal for the whole trajectory.
“on every single action, I'm going to give you a strictly better action that you should have taken instead”
The compute required to be first to achieve something is always much larger than the compute needed to catch up, because followers can use tricks like distillation and good initializations as crutches; AlphaGo Zero used ~3E23 FLOPS as a huge outlier, but Jang reproduced a strong Go bot for roughly $10K of rented compute.
“The compute required to be the first to do something is always much larger than the compute it takes to catch up.”
A value function lets a network look at a board and instantly predict win probability—amortizing a huge number of possible game playouts into a single forward pass—just as humans glance at a board and 'know' they will lose without playing it out; this is the key to truncating search depth in Go.
“a neural network in a human can somehow do all of this simulation at a glance”
In deep learning, initialization is everything: you should always initialize a research project as close to success as possible—starting with something that works (e.g., supervised learning on expert data) and then improving it—rather than starting from something that doesn't work at all and trying to make it work.
“In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible”
AlphaGo is an elegant RL algorithm because it never has to start at a zero-percent success rate or solve the exploration problem of reaching non-zero success; every step is stable supervised learning (value classification plus policy KL minimization) on improved labels, so you can train arbitrarily large networks stably with simple infrastructure and never sit in a flat-gradient local minimum.
“The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate.”
There is a deep duality between MCTS and Q-learning: both propagate value information through a recursive dynamic-programming structure, but MCTS plans over trajectories the agent hasn't visited yet (forward search), while Q-learning plans over trajectories it has already visited—Q-learning mattered historically because we couldn't do forward search on high-dimensional problems like robotics.
“in one case you're planning over trajectories your agent hasn't been to yet, whereas in the other you're planning over trajectories your agent has visited.”
When you cannot easily build a search tree (imperfect-information or non-deterministic games like StarCraft), neural fictitious self-play replaces the MCTS teacher: you fix an opponent, train a best-response policy with a model-free RL algorithm (PPO, SAC, V-MPO), and distill the league of best responses into a mixed strategy—still fundamentally relabeling states with better actions.
“In MCTS, we perform search... In a game where you can't easily simulate a search process, what they do instead is train what's known as a best response policy.”
MCTS is only guaranteed to improve the policy if the value estimates at the leaves are accurate; if training data lacks late-stage resolutions (e.g., bots resign rather than play to Tromp-Taylor end), bad terminal values propagate up through backups and PUCT, making MCTS produce a worse distribution than the raw policy—so the value function must be grounded, e.g. by forcing some games to resolve fully.
“Ultimately, MCTS will fall apart if you don't have a grounding function for the value.”
Naive win-reinforcement self-play RL is extremely high variance because in a matchup between evenly matched policies, only one critical move in one game out of ~100 (each ~300 moves) actually drove the win, so you have one useful supervision signal buried in ~99×300 neutral labels; the gradient variance grows quadratically with trajectory length T.
“you only have one label out of this enormous dataset of supervision actions”
Off-policy training in AlphaGo is fine—and even helpful in a DAgger sense—as long as the replay buffer is mostly states the current policy would visit plus a reasonable band of nearby states that funnel back to the optimal trajectory; it becomes harmful when MCTS relabels states the policy would never reach, since the network then wastes capacity learning good actions on unreachable states.
“if you take this to the extreme... you're training your model on states you would never reach. This is where off-policy can really hurt.”
Distillation is so effective per sample because soft targets (the full logit distribution) have far higher entropy than a one-hot token answer, carrying much more information per sample; AlphaGo exploits this by training the policy network to imitate the full MCTS visit-count distribution rather than just the single selected action ('dark knowledge' distillation).
“There's way more information in bits per sample in a soft label. That's why distillation is so effective per sample.”
AlphaGo Lee averaged the value network estimate with a real policy-vs-policy playout to the end (α·Vθ + (1−α)·rollout) to ground value estimates, but all subsequent papers removed this; Jang's implementation also dropped it, which speeds things up greatly since you don't roll games out on every simulation.
“It turns out this is totally unnecessary. In all subsequent papers after AlphaGo Lee, they got rid of it.”
The benefit of any given algorithmic compute-multiplier is likely transitory: many of KataGo's clever tricks (architecture choices, auxiliary supervision objectives, asynchronous distributed setups) matter much less on faster modern GPUs like Blackwell, and three years out they may stack even worse—so research taste is knowing how much the Bitter Lesson can buy at any given moment.
“Maybe at any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the KataGo paper.”
Individually good research ideas don't necessarily stack—training runs at AI labs have failed because of weird interactions between two seemingly good ideas pursued in parallel—which is why a single top-down vision of how the whole system should work matters, and why compute multipliers that have correlated benefits may not combine.
“people will individually pursue good ideas, but those don't end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas.”
In perfect-information games like Go there exists a Nash equilibrium strategy that does no worse than any other and can be decided from the current state alone, so AlphaGo's choice to ignore temporal history works well because its Nash-equilibrium play appears superhuman; imperfect-information games (poker, Diplomacy, 2v2 Go) instead require modeling temporal history and opponents.
“In perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.”
Policy-gradient RL is inefficient not only because you must roll out whole trajectories for any signal (samples-per-FLOP falls as horizons lengthen), but also in bits-per-sample: with a 100K vocabulary, an untrained model must stumble on the right token ~100K times to get signal, whereas supervised learning's cross-entropy loss extracts -log(p) bits from a single labeled example; you spend nearly all of training in the low-pass-rate regime where RL learns almost nothing per sample.
“this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning.”
AlphaGo's core conceptual breakthrough was using neural networks to make Go's search problem tractable by shrinking both the breadth (a ~361 branching factor) and depth (~300 moves) of an otherwise astronomically large game tree (on the order of 361^300, more than the atoms in the universe), a problem long thought intractable this century.
“AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable.”
By distilling the MCTS-improved distribution back into the raw policy network, you amortize the work that thousands of search simulations do into a single forward pass, so the next round of self-play starts from a much stronger base and reaches higher win rates for the same simulation budget.
“if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here”
Andy Jones's 'Scaling Scaling Laws with Board Games' showed you can trade off test-time compute and training compute—spending more compute on MCTS search yields performance equivalent to more training compute—anticipating LLM inference scaling, and additionally predicted how much compute is needed to solve a larger version of a board game.
“you can trade off test-time compute and training compute. That is to say, you can spend more compute searching through MCTS, and if you do that, you can get the equivalent performance to having spent more time training the model.”
Current frontier coding models excel at hyperparameter optimization over open-ended search spaces and at executing any specified experiment (running, plotting, reporting), but they are not yet good at choosing which experiment to run next in a research track or at the lateral thinking needed to recognize a dead-end track and step back to first principles—researchers still must catch infra bugs and steer.
“the current closed models the public can access today don't seem to be that great at selecting what the next experiment should be in a given track.”
In the fullness of time, compute is the single most important determinant of capability—scaling up energy, compute, and parameters inevitably yields intelligence and no algorithmic detail matters beyond that—but in the present day, finite compute forces researchers to use heuristics, which are likely redundant and transitory.
“in the fullness of time compute is the single most important determinant of how things work.”
A trait of a strong researcher (per Ilya Sutskever) is a strong belief in what the correct idea is, which lets them persevere through bugs—knowing that because the idea should work there must be a bug, rather than concluding the idea is wrong; this raises the question of how locally verifiable a good idea is for an automated researcher.
“He's able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea”
Late-stage Go positions are nearly decidable—with most stones placed there is little tree-depth uncertainty—so games played to the end by reasonable players give good training data for the value function, and the hardest part to learn is who is winning in the mid-game.
“It's quite easy to evaluate a late-stage Go game. When almost all the pieces are on the board, it's almost like a decidable problem”
The optimal RL target discards the moves that did not contribute and gradients only on the one move that actually made you better; advantage estimation (pushing up actions better than average, down those worse than average) approximates this, requiring a good estimate of average performance from a state—bringing back the value function.
“the optimal case is to discard all of these moves, and only get a gradient on that single move that you got better on.”
The reason MCTS is so much more sample-efficient than LLM/robotics RL is that you can locally improve your policy per move—the search, truncated by a working value function, suggests a better next move even without unrolling the whole trajectory or being able to win the game—whereas in normal LLM/robotics RL there is no way to evaluate and improve a move independent of actually solving the whole problem.
“you can do this much more sample-efficient, favorable thing with Go is that because MCTS works in Go, you basically know that if you just do the search locally here...you can just say, 'This is my new policy'”
Although AlphaGo vs. Lee used tens of thousands of simulations per move on a TPU pod, modern Go bots need little test-time compute because the policy improvement progressively pushes the search burden into the raw network, so most work is done in one forward pass with search only adding extra strength.
“modern Go bots don't need that much compute at test time.”
Current LLM RL treats an entire generated sequence as a single action with T=1, summing token log-probs into one log-prob, because decomposing into per-token rewards introduces cross-correlations between actions that magnify gradient variance—this is why LLMs do one-step rather than multi-step RL.
“In current LLM RL, they treat this entire sequence as a single action, at, and big T is just one.”
MCTS works for Go because value estimation is concrete (groundable) and the action space is discrete and bounded, suiting the PUCT exploration heuristic; for LLMs this breaks down because the √N/(1+Na) term assumes you re-sample the same child, but language's vast open-ended action space means you almost never sample the same continuation twice, so PUCT is too greedy and not the right way to guide tree search.
“In an LLM, you're most likely never going to sample the same child more than once.”
DeepMind's experience solving games (Atari, Go, StarCraft) plausibly produced positive transfer—in coding, general research ability, or project management—that now helps them build good LLMs; if true, this suggests automated AI researchers could likewise transfer skills from quick-to-verify environments to ambitious economically useful domains like drug discovery.
“there was some positive transfer from their time working on games and Atari and Go and StarCraft that now helps them make good LLMs.”
Modern RL has largely converged on on-policy setups that don't directly train on off-policy data—at best using off-policy data to reduce variance or shape rewards via advantage computation—because doing so is more stable and prevents off-policy dynamics problems from blowing up the loss.
“Much of RL has converged to a much more on-policy setup where they don't really try to directly train on off-policy data.”
Cryptographic protocols and neural networks are structurally similar—both are sequential layers that jumble information so the output depends on all inputs—a convergent evolution where cryptography wants maximal sensitivity to initial conditions and neural networks want all information considered together, with maximum power at the edge of chaos.
“There's a convergent evolution in the algorithms.”
Although LLMs already do something resembling human reasoning (trying an approach, backing up, trying another) without an explicit tree, the jury is out on the final form of LLM reasoning, and the idea of forward search and simulation to assess what is valuable may make a comeback—even if not in exactly the AlphaGo instantiation.
“I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even if not in exactly the same instantiation as AlphaGo.”
On small boards (4x4 up to 9x9), purely random play over ~50,000 games generates reasonable-looking end states that can be scored easily, yielding a good value function; combined with architectures that transfer between 9x9 and 19x19, this dramatically cuts the warm-start time for tabula rasa learning.
“If you play 50,000 games, you'll actually learn a pretty good value function as well. On a 9x9 board, you can see enough of the common patterns with random play.”
Go is an ideal substrate for training automated AI researchers because it captures many research problems overlapping with LLMs and robotics yet is quick to verify (you can easily check who won), providing a hard-to-reward-hack outer loop while the inner loop exercises research engineering—skills that may then transfer to domains like biosciences, robotics, or AI research itself.
“Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it's very quick to verify.”
You should study scaling laws only after you have a working, bug-free system with good data and architecture; trying to derive scaling laws on a regime where the policy isn't working (e.g., with bugs in MCTS labeling) just means studying scaling laws on bad data—you must first make a strong artifact, then use scaling laws to extrapolate.
“if you're in a regime where your policy is not working well, you might just be studying scaling laws on bad data.”
For Go, ResNets outperform transformers in small-data, low-budget regimes because their convolutional inductive bias captures local features efficiently, while transformers' global attention only wins when more global context and more data are available; KataGo additionally found it useful to pool global features so value can connect across the board.
“For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets.”
Computer Go uses Tromp-Taylor rules because they are completely unambiguous and can be resolved algorithmically by a computer, whereas human Chinese/Japanese scoring relies on both players' value functions agreeing on consensus; under Tromp-Taylor, positions that humans would resolve by judgment must be played out to the end.
“Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against.”
In Go it can be optimal to let an opponent capture some stones if doing so lets you capture more stones elsewhere—you can lose the battle but win the war—and as board size increases these micro-versus-macro dynamics become more interesting.
“In Go, it's actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board.”
Google's seemingly late start in LLMs may actually reflect a longer pre-training on scaling up TPUs—investing their tech tree in making TPUs good, which seemed less useful short-term but pays off long-term—illustrating that even humans struggle to reason about the optimal research strategy with today's data.
“who knows if the seeming late start was really just them pre-training for longer on how to scale up TPUs.”
Once you understand the Go search problem framing and build up the pieces, AlphaGo becomes much more understandable than anticipated.
“It's okay to be confused right now, but it's probably simpler to understand, by the end of this lecture, than you anticipate.”