AutoRegression
A self-optimizing AI framework that autonomously improves algorithms by running constrained experiments, keeping improvements and discarding failures, without human involvement during iteration
As defined in AutoResearch explained..
Other definitions
The mechanism by which language models generate output one token at a time, each conditioned on previous tokens; source of probability-driven biases.
343 | Tom Griffiths on The Laws of ThoughtGenerating output one token at a time, each conditioned on prior tokens, which Grant argues biases models toward predictable continuations.
Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of math
Where it comes up · 3
- AutoResearch explained..14 claims
- Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of math3 claims
- 343 | Tom Griffiths on The Laws of Thought1 claim
Most important claims · 10 of 18Search all claims for “AutoRegression” →
- Raw language models like GPT-4 are very sensitive to the probability of the outputs they must produce: they are better at counting strings of 30 letters than 29 because the number 30 appears more often on the internet than 29, so when nearby high-probability answers exist the model produces the high-probability one rather than the correct one—an idiosyncratic bias the authors called 'embers of autoregression.'343 | Tom Griffiths on The Laws of Thought
- Autoregressive next-token prediction is a strange way to generate mathematics, because the substantive cross-field connections are by nature unlikely next tokens, so training on predictable continuation may suppress them.Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of math
- AutoRegression self-optimized a restaurant inventory algorithm by running constrained experiments without direct human involvement, improving the algorithm to fill inventory at the right time and significantly reducing order failures.AutoResearch explained..
- Modern software development is evolving such that creating software with AutoRegression is less about detailed implementation and more about understanding the problem deeply enough to define it clearly and structure it properly for the agent to build successfully.AutoResearch explained..
- AutoRegression requires human guidance to find the right metrics to optimize and to design proper evaluation functions and structural constraints; without this guidance, it cannot scale beyond narrow, well-defined tasks.AutoResearch explained..
- When direction is vague (e.g., 'make this restaurant better' without clear objectives), AutoRegression cannot scale beyond its narrow task because the lack of specificity prevents proper structuring and metric definition.AutoResearch explained..
- Autoregressive next-token generation is a strange way to produce insight, because the valuable connections are by nature unlikely next tokens, and standard RL does not specifically upweight such low-probability leaps, so the system may hold intelligence locked inside an awkward interface.Dwarkesh Patel Podcast - Grant Sanderson – AI and the future of math
- AutoRegression's internal mechanism runs experiments incrementally, keeping only those that improve the algorithm's evaluation metric while discarding experiments that fail, creating a ratcheting improvement pattern.AutoResearch explained..
- AutoRegression does not work effectively for problems outside narrow, well-defined domains where simulation and clear feedback loops are unavailable.AutoResearch explained..
- AutoRegression learned that lead time for ingredient delivery (3-5 days) means it should place orders on day one rather than waiting for stock depletion, maximizing initial inventory to create a buffer against stockouts.AutoResearch explained..