chain-of-thought
A model generating intermediate reasoning steps before producing a final output or action.
As defined in Fully autonomous robots are much closer than you think – Sergey Levine
Other definitions
Prompting technique that encourages language models to break down complex reasoning into intermediate steps, showing the step-by-step reasoning process rather than jumping directly to answers, improving accuracy on complex tasks
The Engineering Unlocks Behind DeepSeek | YC Decoded
Where it comes up · 5
- Sholto Douglas & Trenton Bricken — How LLMs actually think3 claims
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history2 claims
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history2 claims
- The Engineering Unlocks Behind DeepSeek | YC Decoded1 claim
- Fully autonomous robots are much closer than you think – Sergey Levine1 claim
Most important claims · 9 of 9Search all claims for “chain-of-thought” →
- Chain-of-thought is often not representative of how the model actually decides its answer: you can garble or edit the reasoning and the model still outputs the true answer, or ablate the chain-of-thought entirely and get the same answer, and models will fabricate plausible-sounding but misleading reasoning—so chain-of-thought cannot be trusted as a transparency tool for AI safety.Sholto Douglas & Trenton Bricken — How LLMs actually think
- The π0 model is a vision-language model adapted for motor control: it reads robot sensory input, does internal processing including chain-of-thought intermediate steps, and routes to an action expert (a separate module because actions are continuous and high-frequency) within an end-to-end transformer structured as a mixture-of-experts.Fully autonomous robots are much closer than you think – Sergey Levine
- Through reinforcement learning alone with GRPO and simple reward signals, Deep Seek's R1 model developed reasoning behavior that emerged over thousands of RL steps, learning extended chain-of-thought skills and experiencing 'aha moments' where it recognized its own mistakes and backtracked to correct its reasoning.The Engineering Unlocks Behind DeepSeek | YC Decoded
- The best synthetic data is data that required a lot of reasoning to create and whose reasoning can be verified—which is why DeepMind's geometry work succeeded: geometry is easily formalizable and verifiable, so you can generate heaps of verified proofs and train on them as known-good data, with the real world serving as verifier in other domains.Sholto Douglas & Trenton Bricken — How LLMs actually think
- Chain-of-thought doesn't just transmit the single output token; during the forward pass the model creates KV (key-value) values that future steps attend to, so the model can smush information about potential futures into the keys and values it might want, transmitting far more than the tiny log-vocab-size bits of one token.Sholto Douglas & Trenton Bricken — How LLMs actually think
- RLHF (reinforcement learning from human feedback) unhobbled GPT-2's raw capabilities into a functional chatbot, achieving roughly a 100x improvement in human preference ratings compared to non-RLHF models, and enabled basic chain-of-thought reasoning, but the underlying model still possessed many unutilized raw capabilities that could be further unlocked.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- GPT-4 currently performs chain-of-thought reasoning with a few hundred tokens of test-time compute (equivalent to about 3 minutes of human thinking at 100 tokens per minute), but there exists a massive 'test time compute overhang' where models could theoretically use millions of tokens for iterative debugging and planning, representing a +4 orders of magnitude improvement if unlocked.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- A current alignment advantage is that we can read what models are thinking via legible chain-of-thought (we bootstrap by pre-training them to think in English), so initial automated alignment researchers may not need complicated mechanistic interpretability; but this is likely not the most efficient architecture—a recurrent architecture with internal states would be more efficient—so during a rapid intelligence explosion we risk transitioning within a year from legible, understood systems to alien, illegible architectures, making it an incredibly hairy period for alignment.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Current models only do a few hundred tokens of chain-of-thought before getting stuck, but unlocking 'System 2' coherent thinking over millions of tokens would represent roughly +4 OOMs of test-time compute on a single problem, which—per the AlphaGo train/test tradeoff—is comparable to a 3.5x-OOM bigger model, representing a large untapped overhang.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history