Large language models score 0% accuracy on real-world data warehouse text-to-SQL tasks despite 85% accuracy on public benchmarks (Spider, BIRD) because real data contains domain-specific idiosyncratic data not present in training data, has schema complexity (100+ lines of SQL vs. 10-20 lines) and messy naming conventions (underscores, non-mnemonic column names) that public benchmarks lack.

causalpending

Speaker

Mike Stonebraker

Evidence Quote

“on our benchmarks...large language models get 0%...Data warehouse data is not in the pile...Real world data warehouses it's 100 lines of SQL...column names are often underscore Zuppers...not mmonic”

Source

Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker— Ryan Peterman
Created: 8/12/2026, 6:09:46 PM

Your notes

Loading notes...