HumanEval
A coding benchmark consisting of programming problems used to evaluate code-generation models.
As defined in Francois Chollet — Why the biggest AI models can't solve simple puzzles
Where it comes up · 4
- What's next for AI agentic workflows ft. Andrew Ng of AI Fund2 claims
- Microsoft Goes FULL NUCLEAR to Develop AGI...2 claims
- Francois Chollet — Why the biggest AI models can't solve simple puzzles1 claim
- Sholto Douglas & Trenton Bricken — How LLMs actually think1 claim
Most important claims · 6 of 6Search all claims for “HumanEval” →
- On the HumanEval coding benchmark, GPT-3.5 with zero-shot prompting achieves 48% accuracy, GPT-4 achieves 67% accuracy, but GPT-3.5 wrapped in an agentic workflow actually outperforms GPT-4, and an agentic workflow around GPT-4 performs even better.What's next for AI agentic workflows ft. Andrew Ng of AI Fund
- Apparent emergent abilities are partly a 'mirage' of measurement: success requires sampling the last several tokens correctly, so you multiply those probabilities and below a threshold of reliability you see nothing, then suddenly success—when the underlying ability was smoothly there all along, and using a smooth metric like log pass rates reveals the gradual progress.Sholto Douglas & Trenton Bricken — How LLMs actually think
- The HumanEval benchmark tests coding problems such as given a list of integers, return the sum of all elements at even positions, which humans would solve through an iterative drafting and testing process rather than writing code in a single pass.What's next for AI agentic workflows ft. Andrew Ng of AI Fund
- On the HumanEval coding benchmark, AI models have improved from 32.2% accuracy (pre-September 2021) to 99.39% accuracy (Claude 3.5 Sonnet), a roughly 3x improvement in 3 years.Microsoft Goes FULL NUCLEAR to Develop AGI...
- Grok Moe, at 6.6 billion parameters, scores 74.4% on the HumanEval benchmark, compared to 65% for a 175 billion parameter model in September 2022, representing roughly 2.7x parameter efficiency.Microsoft Goes FULL NUCLEAR to Develop AGI...
- Prior AGI-aspiring benchmarks like MMLU and MATH followed an almost exponential saturation curve—scoring 5%, 10%, 30%, 40% as compute increases across models, then shooting up to near-100%—and on HumanEval, upweighting cases where models occasionally get problems right (1/1000 to 1/100 to 1/10) leads to total saturation, raising the question of why ARC won't follow the same pattern.Francois Chollet — Why the biggest AI models can't solve simple puzzles