On the HumanEval coding benchmark, AI models have improved from 32.2% accuracy (pre-September 2021) to 99.39% accuracy (Claude 3.5 Sonnet), a roughly 3x improvement in 3 years.
factualpending
Speaker
Wes RothEvidence Quote
“before September 2021 the best thing we had looks like had 32.2 accuracy on this human ofal Benchmark and looks like now we have ones that clock in at 99.39% 3 Cloud 3.5 Sonet”
Created: 8/10/2026, 10:52:56 PM
Your notes
Loading notes...