Text-to-SQL Accuracy: Why the Benchmarks Do Not Predict Your Schema

The benchmark numbers, and why you cannot use them

Text-to-SQL vendors quote accuracy from public benchmarks. Spider, BIRD, and their successors report execution accuracy in the 80s and low 90s for frontier models. Those numbers are real and they do not transfer to your warehouse.

Benchmark databases average 5 to 15 tables with descriptive column names and documented relationships. A mid-market retail warehouse runs 200 to 800 tables. The columns are named qty_1, amt_net, and fl_actv, because they were named by an ERP vendor in 2011 and nobody was allowed to rename them.

The benchmark measures whether a model can write SQL. Your problem is whether a model can write SQL about your business. Those are different tasks and only one of them is being scored.

The four things that actually drive accuracy

Schema size, nonlinearly

Accuracy does not decay gently as tables are added. It falls off a cliff once the schema exceeds what fits usefully in context.

Below roughly 20 tables, a good model reasons over the whole schema and picks correctly. Past a few hundred, the system has to retrieve a subset of the schema before it can write anything, and now you have a retrieval problem sitting upstream of the SQL problem. If retrieval hands the model the wrong five tables, the model writes flawless SQL against the wrong five tables.

Most reported production failures are retrieval failures wearing a SQL costume.

Naming quality

This is the largest single lever and it is unglamorous. A column named net_sales_usd gets selected correctly far more often than the identical column named ns_a.

You do not have to rename anything in the source system. You need a description layer: a mapping from physical column to a plain-English name and definition that the model sees instead of, or alongside, the raw name. Teams that invest two weeks in this typically see a larger accuracy gain than they would from a model upgrade.

Join cardinality and fan-out

The classic silent error. Join a transaction table to a table with multiple rows per key and every sum in the query is inflated.

A model that does not know a relationship is one-to-many will happily write the join. The query runs. Revenue reads 2.3x actual. Nobody notices, because nobody was expecting the number to be exactly anything.

Fan-out errors are the most common cause of wrong-but-plausible output in production text-to-SQL, and they are nearly invisible without a reconciliation step.

Temporal logic

Retail runs on a 4-5-4 fiscal calendar. Week 1 does not start January 1. Comparable-store sales exclude stores open less than 13 months. Year-over-year in a 53-week year needs a shifted comparison or the whole thing is off by a week.

None of this is in the schema. A model asked for "comp sales versus last year" will produce a calendar-year comparison against all stores, which is a real number that answers a different question.

How to measure accuracy on your own schema

Execution accuracy, the benchmark metric, means the query ran and returned rows. It is nearly useless to you. Measure these three instead.

Exact-match accuracy. The returned number equals the number a trusted analyst produces. This is the only metric that matters to a business user. Expect it to be 20 to 40 points below whatever the vendor quotes.

Silent-failure rate. Of the queries that executed successfully, what share returned a wrong number with no error and no warning. This is the risk metric. Anything above 10% means the tool cannot be used unsupervised.

Refusal rate. How often the system declines to answer rather than guessing. Counterintuitively, you want this number to be non-zero. A system with a 0% refusal rate is not calibrated, it is confident, and those are opposite properties.

Building the test set

Thirty questions is enough to be informative and small enough that you will actually do it.

Pull them from real requests your analytics team received last quarter, where a human produced a checked answer. Weight the set toward the ugly ones. Ten straightforward aggregations, ten that require a non-obvious join, and ten that depend on business logic that is not in the schema.

Run it quarterly. Model versions change underneath you and accuracy is not monotonic.

What to do about it

In order of return on effort:

1. Write a semantic description layer for the 40 tables that answer 90% of questions. Not all 480. The distribution is steep. Two weeks of work, largest single accuracy gain available.
2. Define your metrics once, centrally. Net sales, comp sales, margin, units. One definition each, owned by a named person. Every downstream tool reads from it.
3. Encode the fiscal calendar as data, in a date dimension table with fiscal week, period, and comp flags. Then it is in the schema and the model can find it.
4. Require cited SQL on every answer. Fan-out errors are catchable in the query text and invisible in the result.
5. Constrain the surface. Route questions to pre-validated query templates where you can, and let free-form generation handle the tail.

Notice that four of the five are data engineering, not AI. The accuracy problem in text-to-SQL is mostly a metadata problem that has been unaddressed for a decade and became visible the moment something tried to read the schema without an analyst's memory attached.

Key takeaways

• Public text-to-SQL benchmarks run on 5 to 15 table databases with clean names. Your warehouse has 200 to 800 tables named by an ERP vendor. Benchmark accuracy in the high 80s does not transfer.
• Accuracy falls off a cliff, not gradually, once the schema exceeds context. Past a few hundred tables you have a retrieval problem upstream of the SQL problem, and most production failures are retrieval failures.
• Column naming is the largest single lever. A description layer mapping physical columns to plain-English definitions usually beats a model upgrade, and takes about two weeks for the 40 tables that answer 90% of questions.
• Join fan-out is the most common silent error. Joining to a one-to-many table inflates every sum, the query runs clean, and the number is wrong by a multiple nobody was positioned to notice.
• Measure exact-match accuracy, silent-failure rate, and refusal rate. Execution accuracy is the benchmark metric and it tells you nothing. A 0% refusal rate means the system is uncalibrated, not good.
• Four of the five highest-return fixes are data engineering, not AI. Text-to-SQL accuracy is mostly a decade-old metadata problem that only became visible when something tried to read the schema without an analyst's memory attached.