On the HumanEval coding benchmark, AI models have improved from 32.2% accuracy (pre-September 2021) to 99.39% accuracy (Claude 3.5 Sonnet), a roughly 3x improvement in 3 years.

factualpending

Speaker

Wes Roth

Evidence Quote

“before September 2021 the best thing we had looks like had 32.2 accuracy on this human ofal Benchmark and looks like now we have ones that clock in at 99.39% 3 Cloud 3.5 Sonet”

Source

Microsoft Goes FULL NUCLEAR to Develop AGI...— Wes Roth
Created: 8/10/2026, 10:52:56 PM

Your notes

Loading notes...