Meta harness
A system that treats agent harness architecture as an optimization target, using agentic proposers to read failed execution traces and automatically generate improved harness designs
As defined in Rethinking AI Agents: The Rise of Harness Engineering
Where it comes up · 1
Most important claims · 6 of 6Search all claims for “Meta harness” →
- Meta harness achieved rank 2 with Opus (Claude's larger model) and rank 1 with Haiku (Claude's smaller model), with the smaller model outranking the larger one through harness optimization alone.Rethinking AI Agents: The Rise of Harness Engineering
- On 215 class text classification, Meta harness achieved 48.6% accuracy, 7.7 points above state of the art, using four times fewer tokens.Rethinking AI Agents: The Rise of Harness Engineering
- Meta harness scored 76.4% on Terminal Bench 2, becoming the only automatically optimized system in a field of hand-engineered entries.Rethinking AI Agents: The Rise of Harness Engineering
- The raw trace details in Meta harness optimization are irreplaceable: removing them drops accuracy from 50% to 34.6%, while replacing with summaries achieves only 34.9%, demonstrating that the signal lives in raw execution details.Rethinking AI Agents: The Rise of Harness Engineering
- Meta harness, from Stanford's Omar Khattab (creator of DSPy), treats the harness as an optimization target by using an agentic proposer (Claude Opus 4.6) that reads failed execution traces, diagnoses breakage, and writes complete new harnesses, with evaluators testing each proposal and repeating the loop, consuming 10 million tokens per iteration (400 times more feedback than prior methods) across 82 files read per round.Rethinking AI Agents: The Rise of Harness Engineering
- Four systems (NLAH representation, Meta harness optimization, DeepMind's auto harness, Agent Spec) represent four complementary facets of harness engineering: representation, optimization, constraints, and safety.Rethinking AI Agents: The Rise of Harness Engineering