Karpathy ran auto-research on NanoChat hyperparameter tuning overnight and it discovered improvements (weight decay on value embeddings, Adam beta tuning) that he had missed despite two decades of manual hyperparameter optimization experience, suggesting joint interactions between parameters that resist human discovery.

normativepending

Speaker

Andre Karpathy

Evidence Quote

“I let auto research go for like overnight and it came back with like tunings that I didn't see”

Source

Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI— No Priors: AI, Machine Learning, Tech, & Startups
Created: 8/12/2026, 6:22:17 PM

Your notes

Loading notes...