How accurate is an LLM connected directly to a database?

Worse than the demo suggests. Anthropic measured Claude at 21% on its own internal analytics questions before building a governed platform around it. On Spider 2.0, a benchmark made of real enterprise warehouse tasks, GPT-4o scores 10.1% and o1-preview 17.1%. The same models clear 85% on older academic benchmarks, which is where the demo feeling comes from.

Why does accuracy collapse on real warehouses?

Real schemas have hundreds of ambiguous fields and no explanation of what the business means by revenue or active. On the BIRD benchmark, removing the hand-written hints that explain field meanings drops GPT-4 from 54.89% to 34.88%. A connector gives the model your tables. It does not give it your definitions, and the definitions are where answers go wrong.

What closed the gap for Anthropic?

Scaffolding, built by their data team: canonical governed datasets, a semantic layer with lineage and business context, skills that encode how a good analyst works, and an eval harness. That moved accuracy from 21% to above 95%. Giving the model raw access to thousands of past correct queries moved it less than one point.

Are LLM database connectors safe for production?

Treat them as read-only ad-hoc tools. Anthropic archived its reference Postgres MCP server, which shipped without query timeouts, row caps, or column blocklists, behind an explicit no-security-guarantees notice. A 2026 audit found it still gets roughly 312k installs a month. Enterprise connectors from the vendors are read-only by policy for the same reason.

What does OptimaFlo do that a connector does not?

It builds the thing the connector assumes. Seven AI roles handle ingestion, Raw to Clean to Ready modeling, orchestration, data quality, dashboards, and a semantic layer, on Apache Iceberg tables in your own cloud. A human approves the SQL before it runs. The chat interface is the last step, and OptimaFlo includes one, running on your choice of Claude, GPT, or Gemini.

Connecting an LLM to Your Data Is the 21% Solution.

I hear the same question in almost every early conversation: why do I need a platform when I can point Claude at Postgres and start asking questions?

It is a fair question. The connectors are real. Claude wires into Snowflake and Postgres, ChatGPT reaches warehouses over MCP bridges, Gemini sits inside BigQuery. The demo is genuinely impressive.

We also now have public numbers on what happens after the demo.

The numbers

Anthropic measured it on their own analytics workload: "Without skills, Claude's ability to answer analytics questions accurately didn't exceed 21% on our evals." Same model you would connect to your database. Their data, their questions, 21%.

The benchmarks agree. Spider 2.0 is built from real enterprise warehouse workflows: thousand-column schemas, hundred-line queries, multiple SQL dialects. GPT-4o scores 10.1% on it. o1-preview reaches 17.1%. The same models clear 85% on the older academic benchmark, which is why the demo feels like magic and the fourth week does not.

BIRD explains the collapse. Every BIRD question ships with a hand-written hint explaining what the fields mean. Remove the hints and GPT-4 falls from 54.89% to 34.88%. Your users do not write hints. They ask "how many active customers do we have" and assume the model knows what active means. It does not. It picks one of your four dozen candidate columns and sums with confidence.

That is the dangerous part. A wrong number does not look wrong. Anthropic names it the silent failure and admits their hardened stack reduces it rather than eliminating it.

What the 74 points cost

Anthropic closed the gap from 21% to above 95%. Not with a bigger model, and not with more data access: they gave the agent grep access to thousands of their own correct past queries and accuracy moved less than one point.

What worked was structure. Governed canonical datasets. A semantic layer where a human owns every metric definition. Lineage. Skills that encode how a careful analyst works. An eval harness watching all of it. Four layers, built and maintained by one of the best data teams in the world. I wrote about who gets to build that scaffolding when the post first landed.

The other vendors built the same conclusion into their products. Snowflake restricts its own MCP server to semantic views because raw schemas lack the meaning an analyst needs. Google wraps Gemini in a Knowledge Catalog and a Looker semantic layer and calls the bundle the product. The connector is the interface; the governed layer underneath is what they sell.

What a connector never builds

Accuracy aside, there is a category error in the pitch. A connector reads tables. Your data problem is that nobody is building the tables.

The connector will not land your sources in a warehouse. It will not model Raw into Clean into Ready. It runs when you open a chat and stops when you close it: ChatGPT sessions drop their state on disconnect and run nothing on a schedule. It watches nothing overnight. It re-learns your business every session, because chat memory is not a semantic layer.

And the plumbing itself is on you. Anthropic archived its reference Postgres MCP server, the one without query timeouts, row caps, or column blocklists, behind a plain warning: no security guarantees. It still gets about 312k installs a month, mostly from people wiring up exactly the setup in the pitch.

Where I land

OptimaFlo runs on these same models. You bring your own Claude, GPT, or Gemini key. I have no argument with the engine.

What we sell is the 74 points: the governed datasets, the semantic layer, the pipelines, the quality checks, and the eval discipline that Anthropic staffed a team to build, shipped as a product one data owner can run, on open Iceberg tables in your own cloud. A human approves the SQL before it touches your tables. The chat interface is in there too. It is the last step, and it is the easy one.

If your warehouse is already modeled and governed, a $20 chat seat on top of it is a great deal, and I say so in our side-by-side comparison. If nobody is building that warehouse, the seat has nothing to stand on.

The 21% is what the model brings. The rest is the job. Come see how we do it, or kick the tires in a 7-day pilot.

Connecting an LLM to Your Data Is the 21% Solution.

The numbers

What the 74 points cost

What a connector never builds

Where I land

We value your privacy