rubric
A structured scoring guide specifying criteria that reward or penalize a model's response, used to grade outputs in a consistent, measurable way.
As defined in Brendan Foody
Where it comes up · 4
- State-Of-The-Art Prompting For AI Agents5 claims
- Brendan Foody3 claims
- Claudia Hauer on War, Education, and Strategic Humanism2 claims
- Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)2 claims
Most important claims · 10 of 12Search all claims for “rubric” →
- When taste cannot be captured in a rubric—as Kant argued—an alternative is RLHF: have the model generate two responses, have people with taste choose which they prefer, and repeat many times until the model understands their preferences.Brendan Foody
- There are two kinds of data: output/curriculum data the model reads and learns from, and measurement data (rubrics, test questions and answers, unit tests) that lets models attempt a problem many times, score responses, and learn from them—and the second kind is the most valuable.Brendan Foody
- Best practice for scoring output from LLMs is to provide detailed rubrics explaining how to differentiate between adjacent scores (e.g., 80 vs 90), though these rubrics are never perfect and always have exceptions.State-Of-The-Art Prompting For AI Agents
- APEX measures economic value by surveying hundreds of experts within each industry about how they spend their time, then having them write prompts and rubrics for each activity bucket; expert time allocation is the best proxy available for economic value because it maps to what customers are willing to pay for.Brendan Foody
- Paradoxically, having teachers make open-ended comparative judgments of student writing with no rubric produces very high agreement (objectivity), while marking the same essays with a detailed rubric and tick-lists produces low agreement (subjectivity), because comparative judgment captures tacit knowledge of quality that prose rubrics fail to specify.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Gemini 2.5 Pro's ability to reason about exceptions to rubrics is valuable for investor evaluation because some investors might be slow to respond but have great track records, and a rigid rubric would penalize them when they deserve a 91 instead of 89.State-Of-The-Art Prompting For AI Agents
- YC has used LLMs to help founders decide which investors to take money from by creating scoring rubrics (0-100 scale) and testing models like GPT-4o Turbo versus Gemini 2.5 to understand how different models interpret and apply the same rubric.State-Of-The-Art Prompting For AI Agents
- War is a combination of method and madness; while readiness can be partly pre-packaged in rubrics, the madness side—what to do when the world around you is unintelligible chaos—can only be confronted through exposure to the liberal arts.Claudia Hauer on War, Education, and Strategic Humanism
- Defining writing quality in words creates distortions, as when rewarding 'fronted adverbials' leads children to write nonsense like 'Forgettably, he crept through the darkness' and earn higher marks than genuinely better writing that lacks the rewarded feature -- the same incentive distortion seen when defenders tuck hands behind their backs because brushing the ball gives a penalty.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Just as there is no rubric or algorithm that works in the fog of war, there is a fog of life as well; the human desire for life hacks and tricks to solve moral dilemmas and dangers is unsatisfiable, and the liberal arts function to sensitize us to the reality that no foolproof algorithm exists for resolving the situations we find ourselves in.Claudia Hauer on War, Education, and Strategic Humanism