Conversational Analytics: What Breaks When You Point an LLM at a Warehouse

What conversational analytics actually is

Conversational analytics means asking a question in plain English and getting an answer computed from your own data. "Which stores are down on margin this month?" returns a list, not a chart you then have to read.

The category also travels under conversational BI, natural language BI, and chat-based analytics. The mechanism is the same in all of them. A language model translates your question into a query, something executes the query against a warehouse, and a second pass turns the result set back into a sentence.

The demo always works. The demo runs on five clean tables with names like orders and customers. Your warehouse has 480 tables, and the margin column is called gm_amt_net_adj.

The three things that break in production

Schema ambiguity

The model has to guess which column you meant. In a real retail warehouse there are typically four columns that could plausibly be "revenue": gross sales, net sales after returns, sales excluding tax, and the finance-approved figure that reconciles to the general ledger.

All four are numerically different. All four are defensible answers to "what were sales." The model picks one, usually the one with the friendliest column name, and reports it without noting that a choice was made.

This is the failure that does the most damage, because the output looks fine. A number came back. It is the right order of magnitude. Nobody catches it until someone reconciles against finance six weeks later.

Join path explosion

Ask for sales by store by category and the model has to find a path from the transaction table to the store dimension to the product hierarchy. In a normalized warehouse there are often three or four valid paths, and they do not return the same rows.

One path goes through the fulfillment table and drops transactions that were shipped from a different location. Another goes through the store dimension directly and includes closed stores. The row counts differ by 6%, which is small enough to look like rounding and large enough to change a decision.

Business logic that lives nowhere in the schema

This is the one nobody budgets for. Your warehouse does not know that:

• Fiscal week 1 starts on the Sunday nearest February 1, not January 1.
• Store 4471 has been closed since March but still receives inventory transfers, so it appears in the data with zero sales and non-zero stock.
• Employee purchases carry a transaction type that finance excludes from comp sales and merchandising includes.
• The three SKUs from the discontinued private label line were re-keyed in April, so year-over-year comparisons on them are meaningless.

Every one of those rules lives in an analyst's head or in a 400-line stored procedure nobody has opened since 2023. A language model reading your schema has no way to infer them.

Confidently wrong is worse than broken

A conversational analytics tool that throws an error is annoying. A tool that returns a plausible wrong number is dangerous, because it moves through the organization at the speed of a Slack message.

Traditional BI had a slow, ugly safety property. Building a dashboard took two weeks and an analyst, and the analyst knew about store 4471. The friction was expensive and it caught things.

Remove the friction without replacing the checking and you have not made analytics faster. You have made wrong answers faster.

The measurable version of this: in internal testing across mid-market retail warehouses, a general-purpose model pointed at a raw schema produces a query that executes successfully about 80 to 90% of the time, and a query that returns the number a human analyst would have produced roughly 40 to 60% of the time. The gap between "it ran" and "it is right" is where the entire problem lives.

What closes the gap

Four things, in order of how much they matter.

1. A governed metric layer, not raw schema access. The model should not be choosing between four revenue columns. It should be selecting from a defined set of metrics where "net sales" has exactly one definition, written down once, owned by a named person. This is the single highest-impact control and it is also the one most vendors skip, because building it is real work and pointing a model at a schema is a weekend.

2. The SQL shown, every time. If the answer arrives without the query that produced it, nobody can audit it, and within a month nobody tries. Showing the SQL is not a power-user feature. It is the mechanism by which a wrong answer gets caught by the one person in the room who knows about the fiscal calendar.

3. Bounded scope. A system that answers 40 well-defined questions correctly is more useful than one that attempts any question at 55% accuracy. Users calibrate fast. They will trust a narrow tool and abandon a broad unreliable one, and the abandonment is permanent.

4. Read-only by default. Separate the question-answering from anything that writes. The failure modes above are tolerable when the worst case is a wrong number on a screen. They are not tolerable when the worst case is a purchase order.

How to evaluate one

Ask for a trial on your own schema, not their demo data. Then run this test.

Pick ten questions your team asked last quarter where you already know the correct answer, because an analyst produced it and it was checked. Include at least three that depend on business logic not present in the schema. Ask the tool all ten.

Score two things separately: how many returned an answer, and how many returned the right answer. Vendors report the first number. The second is the one that predicts whether anyone will still be using the tool in six months.

If the vendor will not run this on your data before a contract, that is the answer to a different question.

Key takeaways

• Conversational analytics fails in production for three reasons: ambiguous schema, multiple valid join paths that return different rows, and business logic that exists only in analysts' heads.
• A general model pointed at a raw warehouse schema executes successfully 80 to 90% of the time and is actually correct 40 to 60% of the time. The gap between those numbers is the whole problem.
• Confidently wrong beats broken as a risk. Traditional BI's two-week build cycle was slow, but the analyst in the loop caught the fiscal calendar and the closed store. Removing friction without replacing the checking just makes wrong answers travel faster.
• A governed metric layer matters more than model quality. If the system is choosing between four columns that could be "revenue," a better model picks wrong more fluently.
• Show the SQL every time. It is not a power-user feature, it is the audit mechanism, and without it errors stop being catchable within about a month.
• Evaluate on your own schema with ten questions you already know the answer to. Score "returned an answer" and "returned the right answer" separately. Vendors quote the first.