text-to-SQL
A capability where natural language questions asked by users are automatically translated into SQL database queries; enables non-technical people to interrogate databases without needing to write code or engage data engineers
As defined in Frank Slootman on Focus, Performance, Bureaucracy, & Success | The Knowledge Project Podcast 173
Where it comes up · 2
- Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker4 claims
- Frank Slootman on Focus, Performance, Bureaucracy, & Success | The Knowledge Project Podcast 1731 claim
Most important claims · 5 of 5Search all claims for “text-to-SQL” →
- Natural language interfaces and text-to-SQL are becoming table stakes; they make data interrogation much easier than dashboards for most use cases; this enables more people to ask questions of data without needing data engineers or analysts.Frank Slootman on Focus, Performance, Bureaucracy, & Success | The Knowledge Project Podcast 173
- Large language models score 0% accuracy on real-world data warehouse text-to-SQL tasks despite 85% accuracy on public benchmarks (Spider, BIRD) because real data contains domain-specific idiosyncratic data not present in training data, has schema complexity (100+ lines of SQL vs. 10-20 lines) and messy naming conventions (underscores, non-mnemonic column names) that public benchmarks lack.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- For text-to-SQL systems to be viable, they must provide the retrieval system with complete context (FROM clause and all JOIN terms) and break complex queries into simpler pieces, rather than expecting LLMs to reconstruct this information from schema alone.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Human SQL programmers with knowledge of the schema achieve very high accuracy (90% or better) on the Beaver text-to-SQL benchmark, establishing a human baseline that LLMs fail to approach.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Stonebraker published a benchmark called Beaver, which is an anonymized and abstracted version of real production data warehouse schemas from four organizations, designed to provide a more realistic test of text-to-SQL systems than public benchmarks like Spider and Bird.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker