normative

Benchmarks Are Useful But Not Substance

Benchmarks and tools like Chatbot Arena—which pits models head-to-head and lets users judge which answer is better, accumulating sports-like scores via the same human-reinforcement-learning A/B mechanism that trains the models—are useful and entertaining indicators of capability, but one can be overly rotated on them; they are not the substance of what AI will mean for our lives.

normativepending

Speaker

Reid Hoffman

Evidence Quote

“you can be overly rotated on them. It's, like, 'Aha. Mine was Number One on these 10 things.' And, you're like, 'Well, yes, that's good.'”

Source

Why AI Is Good for Humans (with Reid Hoffman)— EconTalk
Created: 6/13/2026, 12:18:02 AM

Your notes

Loading notes...