ChatGPT for data analysis: what it does well, and where it breaks

Advanced Data Analysis is not overhyped for what it actually is: hand it a file, and it runs real Python, computes real numbers, and charts what you ask for, in one sitting. The gap shows up in three specific, structural places, not in the quality of any single answer.

No persistence between sessions

Close the conversation and open a new one, and the file, the dataframe it built, and the context you established are gone. Every session starts cold. For a single analysis that's a minor inconvenience: re-upload, re-ask. For anything you'd want to check on a recurring basis, weekly revenue, a running quality metric, it means a person has to remember to repeat the whole ritual every time, because nothing persists on its own between conversations.

No scheduling or triggers

Nothing runs unless a person opens a chat and asks. There's no equivalent of a pipeline that refreshes on a cron schedule, or an alert that fires when a number moves outside its normal range. Whatever monitoring exists, exists because someone remembered to go look.

Accuracy drops hard on real warehouse tasks

General chat quality is not the same benchmark as "read our actual, messy, real-world schema and answer a business question correctly." On Spider 2.0, built specifically from real enterprise warehouse workflows rather than clean academic examples, GPT-4o scores 10.1% and o1-preview scores 17.1%, against 86.6% for the same class of model on the older, easier Spider 1.0 benchmark. Anthropic reports its own internal analytics agent scored 21% before the company built a governed platform around it: canonical datasets, a semantic layer, encoded analyst expertise, and a validation harness, after which accuracy went above 95%.

The mechanism behind the gap is context a model doesn't have by default. On the BIRD benchmark, stripping out the hand-written hints that explain what fields actually mean drops GPT-4's accuracy from 54.89% to 34.88%. Real users don't supply those hints when they type a question; they just ask, the same way they'd ask a colleague who already knows the business.

Why this isn't really a ChatGPT problem

Every point above generalizes across chat LLMs, ChatGPT, Claude, Gemini, because none of them ship with your business's context built in. The fix Anthropic published for its own product is the same shape of fix any of them needs: governed data, a semantic layer, encoded expertise, and validation, which is infrastructure work, not a better prompt.

Don't choose OptimaFlo if

None of this means Advanced Data Analysis is a bad tool. It's a genuinely good one for what it's built for.

• Your questions are about a file, not a governed warehouse. Keep using it. Building infrastructure around a single spreadsheet is over-engineering a problem that doesn't need it.
• You already have the four layers Anthropic describes. A modeled warehouse, a semantic layer, and someone checking answers means a chat seat on top is a bargain, not a gap.
• You need general reasoning help, not a governed data platform. ChatGPT is excellent at that job. This page is specifically about the warehouse-accuracy gap, not a verdict on the model overall.