Karpathy ran auto-research on NanoChat hyperparameter tuning overnight and it discovered improvements (weight decay on value embeddings, Adam beta tuning) that he had missed despite two decades of manual hyperparameter optimization experience, suggesting joint interactions between parameters that resist human discovery.
normativepending
Speaker
Andre KarpathyEvidence Quote
“I let auto research go for like overnight and it came back with like tunings that I didn't see”
Source
Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI— No Priors: AI, Machine Learning, Tech, & StartupsCreated: 8/12/2026, 6:22:17 PM
Your notes
Loading notes...