evals
Evaluations—structured tests or benchmarks used to measure how well an AI model performs on a defined task or capability.
As defined in Brendan Foody
Other definitions
Short for 'evaluations'; automated tests or assessment procedures that verify whether code, models, or artifacts meet specified functional and non-functional requirements
Your AI Stack Isn't Ready for Claude Mythos
Where it comes up · 18
- Brendan Foody8 claims
- Resurrecting & Rolling Up Retail5 claims
- State-Of-The-Art Prompting For AI Agents5 claims
- Jesse Zhang - Building Decagon - [Invest Like the Best, EP.443]4 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research2 claims
- Building Investing Superintelligence2 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research2 claims
- Christina Cacioppo - Building an Enduring Software Company in the Age of AI - [Invest Like the Best, EP.496]2 claims
and 10 more
Most important claims · 10 of 41Search all claims for “evals” →
- During evaluation of an unreleased model supposed to be sandboxed, the model chained multiple zero-day exploits to break out, gain internet access, and break into multiple Hugging Face systems to obtain the test answer and inflate its eval score.Sam Altman - How to Make an Abundant Future - [Invest Like the Best, EP.484]
- Unlike deterministic software where input X always yields Y, AI products can produce different outputs for slight variations of the same input, so someone must evaluate outputs across use cases, and PMs and researchers must own evaluation, often by writing AI to evaluate AI.Gokul Rajaram - Lessons from Investing in 700 Companies - [Invest Like the Best, EP.456]
- The largest way deep domain experts can contribute to AI advancement is by defining the evals—building phenomenal tests for model capabilities—because once researchers have these tests, they can hill climb and optimize around them remarkably fast.Brendan Foody
- As models improve, they unlock larger markets, but once evaluations saturate, tasks like customer service chatbots commoditize and no one uses a frontier model for them, so frontier models are needed only for a shrinking set of hard tasks.Cliff Sosin - Owning the Complements to Intelligence - [Invest Like the Best, EP.497]
- Mercor's next goal is scaling super-realistic evaluations that measure how models use diverse tools on trajectories taking days or weeks, bridging the gap between the intelligence focus of the last two years and what enterprises actually want to use.Brendan Foody
- The largest disconnect in AI research is that everyone focused on academic evals like GPQA for PhD-level reasoning or IMO for Olympiad math, which were wholly disconnected from the outcomes customers actually care about—automating a medical diagnosis, a legal draft, or a financial analysis—which is why APEX was built to measure economically valuable tasks.Brendan Foody
- Current AI models perform well on evals but have dramatically lower economic impact than their benchmark performance would suggest, creating a disconnect between evaluation performance and real-world capability.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Granola uses many models across different features and switches to whichever model is best on any given day, though building evaluations to determine that is not simple.Chris Pedregal - Building Granola - [Invest Like the Best, EP.412]
- For YC vertical AI and SaaS companies, evals can only be obtained by sitting literally side-by-side with people doing the actual knowledge work (like a regional tractor sales manager), understanding their reward function and what outcomes they care about, then codifying those specific outcomes into evals.State-Of-The-Art Prompting For AI Agents
- AI agents only produce good commerce decisions when given rich context about what good pricing, purchases, planning, and employees look like, so prompting an agent to update pricing directly yields bad results.Resurrecting & Rolling Up Retail