benchmark
A standardized test or task suite used to measure and compare AI model capabilities.
As defined in An audio version of my blog post, Thoughts on AI progress (Dec 2025)
Other definitions
A reference price used as a standard for comparison. In oil markets, benchmarks like Brent or WTI serve as price references that physical oil deals are negotiated against, rather than as independent bilateral prices.
Oil Markets Expert Explains Global Fossil Fuel Addiction
Where it comes up · 38
- Passive Investing Has Had a Major Impact on the Market: Here is What It Means for Your Portfolio7 claims
- Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots5 claims
- The future of intelligence | Demis Hassabis (Co-founder and CEO of DeepMind)5 claims
- Cliff Asness: A Brief and Biased Survey of Quantitative Investing4 claims
- A $33 Billion Value Manager Who Has Actually Outperformed | Scott McBride4 claims
- Why Money Isn't What You Think: Barry Ritholtz on the Meaning of Money and How NOT to Invest It3 claims
- The Engineering Unlocks Behind DeepSeek | YC Decoded3 claims
- How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes3 claims
and 30 more
Most important claims · 10 of 78Search all claims for “benchmark” →
- Active managers cannot beat a benchmark when money flows out of active funds into the ETF that is the benchmark, because the flow mechanics mean the active fund loses regardless of stock selection, unless it takes concentrated positions a mutual fund cannot hold.The Big Short and Beyond
- Benchmarks weighted by total debt give the largest weights to the most indebted issuers, increasing exposure to default risk, which is a structural flaw compared with equity indexes weighted by market cap.High Yield Credit Investing
- Active share is a quirky metric that is highly sensitive to the benchmark chosen, so different benchmarks yield different active share numbers and can lead to different conclusions about the same security, as AQR research has shown.The Prospects for Active Management
- ICE offers Brent contracts similar to CME's WTI products, but volumes stay concentrated where liquidity is, so ICE's competing contracts see little notable volume, and hedgers benchmark to the market where liquidity exists.CME Group: The House Always Wins
- OpenAI partnered with Khan Academy for two strategic reasons: (1) to benchmark GPT-4's knowledge on standardized tests like the SAT and AP exams where Khan Academy has tens of thousands of test items, and (2) to launch GPT-4 with a handful of organizations that could demonstrate positive social use cases.AI & GPT-4 Revolutionize Education With Sal Khan | EP #35 Moonshots and Mindsets
- There is no single 'price of oil' that people hear about in the news; rather, there are hundreds of different prices for different oil types and regions (WTI, Brent, Naphtha in Europe vs Mediterranean vs Singapore, gasoline, diesel, etc.).Oil Markets Expert Explains Global Fossil Fuel Addiction
- Model benchmarks are not perfectly correlated with real-world utility; factors like product quality, writing style, deployment integration, and business model matter more than benchmark performance for determining market success; for example, Anthropic's Claude is often preferred by some users over OpenAI's models despite lower benchmark scores because of writing style preferences.Week 1 of the Trump administration, AGI timelines, and DeepSeek
- DeepSeek trained its models using H800 chips (not H100s as sometimes reported), which are the result of failed October 2022 export controls that allowed Nvidia to produce degraded but still cutting-edge chips for the Chinese market.Week 1 of the Trump administration, AGI timelines, and DeepSeek
- When frontier models improve at benchmarks, we should attribute it not just to increased scale and clever ML ideas but to billions of dollars spent paying PhDs, MDs, and other experts to write questions and example answers targeting those precise capabilities.An audio version of my blog post, Thoughts on AI progress (Dec 2025)
- Choosing a benchmark, such as the S&P 500 versus MSCI ACWI or the Russell 2000, is itself an allocation decision that drives performance, so it is very tricky to adjust benchmarks each year without rewarding the manager's allocation choices.The Bet With Buffett