Your analytics copilot will be exactly as good as your metric definitions, and no better. That is the finding underneath one of the few sets of hard numbers a named enterprise has published on agentic analytics — and it moves the line item you should be funding this quarter.
Grab, which runs ride-hailing, delivery and financial services across 900-plus cities in eight Southeast Asian countries, published a five-month before-and-after series on its analytics agents on 1 August 2026, and InfoQ wrote it up independently on 17 August. The result is neither the 80% deflection number your vendor is quoting nor the pilot-failure story the survey circuit keeps selling. It is a 14-point shift in what analysts spend their day on, bought with a certified metric layer that took years to build and that no copilot ships with.
What Grab Measured, February to June
The headline number is a shift in the composition of analyst work, not a reduction in analysts. In February, 44% of the tickets Grab's analysts closed were mechanical — the post's own parenthetical is "data preparation, alerting, reporting". By June that share had fallen to 30%. Cycle times fell about 33% over the same window.
The self-serve channel numbers are sharper because they isolate the agent. In March an analyst had to step into half of the threads in Grab's Slack request channel; by May it was under a quarter. Broken out by request type over those two months, the share answered autonomously moved from 53% to 67% for metric questions, 63% to 90% for data pulls, and 50% to 81% for SQL requests. Just under three in four of those threads were started by people outside the analytics team, and 85% got a first response inside one minute.
Two things about that series are worth more than the numbers themselves. First, the unit is Grab's own internal ticket flow rather than a vendor-supplied deflection rate reverse-engineered from a chat log — though Grab publishes no absolute ticket or headcount numbers, so every share here floats free of a denominator you can inspect. Second, the trajectory is legible: data pulls, the most mechanically specified request type, went furthest and fastest; metric questions, which require knowing what a metric means, moved least. That ordering is the whole argument.
Grab also reports what the freed capacity bought. Its BriX data engineering team shipped 31 production deployments, 283 merge requests and 60 features in the first half of 2026, and usage of the BriX portal grew roughly tenfold since last September. The mechanical share did not vanish into a headcount line. It moved into building the surfaces that let stakeholders answer their own questions — the same substitution Group 1 Automotive made when it restructured its F&I desk, and the same one Airbnb's engineering velocity numbers describe.
The Constraint Is Metric Context, Not Model Capability
Grab names its own bottleneck, and it is not the model. "Context sets an agent's ceiling," the engineering post says. "An agent that does not know a metric's grain, its exclusions, and its caveats will guess and confidently produce wrong outputs at speed and at scale." Elsewhere, more bluntly: "Ambiguous definitions cause most analytics slop."
A certified metric is a business measure with one published definition — its grain, its filters, its exclusions and its known caveats — stored somewhere both a human and an agent can read before querying. That is the artifact. Not a dashboard, not a prompt, not a vector index of your Confluence.
Grab built the plumbing to keep it alive. Its ContextIQ platform treats context as something with a lifecycle rather than a document somebody wrote once: it updates automatically when instrumentation specs change, and it patches context documents when an agent fails in production. InfoQ reports the scale as more than 5,000 certified tables and metrics, 4,000 context documents and 2,000 golden records.
That is the invoice nobody puts in the copilot business case. Grab's Analytics Data Warehouse team was already supporting 1,000-plus internal users across 15,000-plus tables and roughly half of all data lake queries before any of this — and was burning about 40% of its time, some two days a week, on repetitive requests. The agents did not create that catalogue. They monetised it.
What the Benchmarks Say About That Ceiling
Independent evaluation says the same thing with a much larger gap, and this is the number to take into your vendor meeting. On Spider 2.0, a 632-task benchmark built from real enterprise warehouses on BigQuery and Snowflake, a code-agent framework running o1-preview solved 21.3% of tasks — against 91.2% on the original Spider and 73.0% on BIRD. Same models. Real schemas with over a thousand columns instead of tidy academic ones.
Now add a semantic layer. A June 2026 paper describes an agent that reasons over curated semantic models rather than raw tables — dimensions, measures, metrics and a declared join graph — and compiles its output deterministically to SQL. It reports 94.15% execution accuracy on the 547-task Spider2-snow benchmark, 515 of 547 correct, against 23.58% for a schema-only agent on the same benchmark.
Treat that spread as an upper bound rather than a measurement. The 94.15% run uses Gemini 3 Pro; the 23.58% baseline is o1-preview, two model generations back — so a model upgrade is folded into the same number as the semantic layer, and the paper runs no ablation separating them. Its own strongest non-semantic baseline, ReFoRCE on the same o1-preview, already reaches 31.26%. Two of the three authors work for DAQUV Corp., which builds the system being scored, and the entry sits third on the public leaderboard behind two systems above 96%. The authors are explicit that the result reflects the semantic layer plus the agent, not the model alone.
The vendors agree, which is the part worth noticing. Snowflake's own engineering blog reported that GPT-4o, which clears 90%+ on Spider, dropped to 51% on Snowflake's internal real-world evaluation set, and that Snowflake Cortex AI's Cortex Analyst reaches 90%+ only as an agentic system coupled with a semantic model. A follow-up post ran the cleanest version of the experiment: the same Claude 3.5 Sonnet went from 57% to 78% average accuracy across four BIRD-SQL domains when the semantic model was supplied — 21 points from the layer, zero from the model.
Read those three together, and take the figure from the only experiment that holds the model fixed: Snowflake's 21 points. That is the defensible number for what a semantic layer buys. The seventy-point spreads have a model upgrade folded into them, and the tempting mirror-image claim — that model choice barely matters — does not survive its own benchmark either: on the Spider 2.0 leaderboard the same Spider-Agent framework runs from 5.5% on Qwen2.5-Coder-32B to 25.8% on Claude 4 Sonnet, about twenty points of spread from the model alone.
So the two levers are closer in size than the semantic-layer pitch implies. What separates them is not magnitude but ownership: the model improves on someone else's roadmap and can be swapped in an afternoon, while the metric layer improves only if you fund it and is worth nothing to a competitor. A bake-off scoring Gemini against Claude on your warehouse is therefore half a decision rather than a wrong one — and the missing half is the metric layer underneath, which no bake-off measures because every vendor demos against a schema someone already curated.
The Autonomy Ladder Is a Governance Artifact
Grab's five-level autonomy model is the part most likely to survive contact with your organisation, because it is a permissions scheme rather than a maturity poster. At L2 the human owns and executes every step and the agent drafts SQL or suggests a visualisation. At L3 the human frames the question and reviews the evidence while the agent writes queries, runs sanity checks and drafts the write-up. At L4 the agent plans and orchestrates end to end, and the human sets intent and guardrails and reviews at defined gates — anomaly, novel scope, sensitive cut. At L5 the agent runs the loop and the human reviews exceptions only.
Crucially, the ladder is per-workflow, not per-company. Grab's Spartan system — a Slack-embedded workflow with more than 50 skills and 120 analysis frameworks — runs the natural-language request path. Scarlet handles near-self-healing pipeline orchestration at L4. Insights Lab is the general-purpose surface for metric, breakdown and root-cause questions in natural language. Different surfaces sit at different rungs.
What stays human is stated explicitly and is the most portable sentence in the whole post: Grab keeps problem framing, canonical metric definitions, the causal story behind a move, business-case assumptions, the go/no-go, and the stakeholder relationship. Notice that canonical metric definitions are on the human list at every level, including L5. The thing the agent is most dependent on is the thing it is never allowed to decide. That is the same separation the governed-autonomy research found in the minority of enterprises that got agents into production.
The underlying stack is unremarkable and that is reassuring. Grab's earlier support-agent write-up names a LangGraph-based workflow engine combined with FastAPI services, with a supervisor agent routing to specialised workflows. Nothing there is exotic; LangGraph is the same orchestration choice most production teams land on. The differentiation is entirely in the context layer.
The Case Against Reading Too Much Into This
The strongest objection is selection. Grab is a data-native platform company that had already certified thousands of tables and metrics before it deployed a single agent. Its analysts write SQL, its stakeholders are technical, and its instrumentation specs are machine-readable enough that context can be regenerated automatically. A regional insurer with metrics defined in nine spreadsheets and one retiring director's head will not see a 14-point shift in five months, and should not budget as if it will.
The second objection is that the benchmark gap may narrow on its own. Model capability on structured data is improving, and the Spider 2.0 numbers cited above use o1-preview, not a 2026 frontier model. If the raw-schema number climbs, the semantic layer's marginal value falls. That is a real possibility and worth re-measuring each quarter rather than assuming.
The third is that Grab is grading its own homework. The metric series is self-reported, the ticket taxonomy is Grab's own, and "mechanical" is a category Grab defines. The post also presents a timeline rather than a controlled comparison — it never claims the agents caused the shift, and a five-month window at a company this size contains reorganisations, hiring and tooling changes it does not account for. InfoQ's write-up is independent reporting on a company post, not an independent audit. Take the direction and the ordering as evidence; treat the exact percentages as a company's account of itself.
The fourth applies to the benchmark evidence: almost everyone sizing the fix sells it. Snowflake sells Cortex Analyst, DAQUV sells the system in the arXiv paper, and only the Spider 2.0 result — which establishes the problem rather than the remedy — comes from a disinterested source. That does not make the vendor numbers useless, and Snowflake's is a genuine same-model ablation. It does mean the honest figure is its 21 points rather than the seventy-point spread from the party with the most to gain.
None of that undoes the core claim, because the core claim is corroborated from directions that fail differently — one operator's five-month series, one academic benchmark establishing how far real warehouse schemas sit from tidy ones, and one controlled vendor ablation, all placing meaningful lift in the context layer. Gartner's finding that AI winners spend roughly 4x more on data than on models is the same result from a fourth. And Forrester's 2026 read that three-quarters of enterprises are chasing agentic AI while few have it in meaningful production is what the gap looks like in aggregate.
What to Do Before You Sign the Copilot Contract
The decision in front of most data leaders this quarter is not which analytics agent to buy. It is whether the next dollar goes to the agent or to the layer underneath it. The evidence points to the layer — not because it is the bigger lever, but because it is the one that stays yours — and buying intent is pointing the same way: Futurum's 1H 2026 survey of 818 decision-makers at organisations above $100M in revenue found 44.5% planning to increase semantic-layer spend over the next 24 months and 14.4% planning to adopt one for the first time — about 59% intending to direct incremental budget there, with only 6.1% ruling it out. Read that as stated intent rather than booked spend; in every category, survey intent runs ahead of the invoices.
This Week:
- Pick your ten most-requested metrics and try to produce, for each, a written grain, filter set, exclusion list and known caveat. Time-box it to two hours. However many you cannot complete is your actual ceiling, expressed as a number you can show a CFO.
- Run your candidate copilot against five questions whose correct answer you already know and whose definitions are genuinely contested internally — not five demo questions. Score exact-match on the number, not plausibility of the SQL.
- Instrument request type in your intake channel today. Grab's series is only readable because data pulls, SQL requests and metric questions were counted separately. Without that split you will get one blended deflection number that tells you nothing about where the ceiling sits.
This Month:
- Put a price on the layer. dbt's published pricing is $100 per user per month on Starter with 5,000 queried metrics included and 20,000 on Enterprise; Cortex Analyst is bundled into Snowflake consumption. Whatever you choose, the semantic layer is now a metered line item and belongs in the same business case as the agent, not a footnote to it.
- Write your autonomy ladder before you need it. Four rungs, per workflow, each with a named gate and a named human owner. Copy Grab's split of what stays human — framing, definitions, causal story, assumptions, go/no-go, relationship — and make canonical metric definitions non-delegable at every rung.
- Decide who owns context maintenance and fund it as an ongoing role. ContextIQ's defining feature is that context updates when instrumentation changes and gets patched when an agent fails. A context layer nobody owns rots inside two quarters and takes the agent's accuracy with it.
Before Renewal:
- Make the vendor state, in the contract or in writing, what accuracy it claims without a curated semantic layer, on your schema. If the answer is a Spider-class benchmark number, you now know it is off by a factor of four on real enterprise warehouses.
- Re-run the ten-metric exercise from step 1 and compare. That delta — not deflection rate, not seats, not tokens — is the number that predicts whether next year's agent investment returns anything.
The Bottom Line
Every platform cycle has a moment where the expensive, unglamorous substrate turns out to be the whole product. Data warehousing had it when companies discovered the ETL cost more than the appliance. Machine learning had it when feature stores ate the model budget. Agentic analytics is having it now, and Grab's five months of numbers are the clearest published account of what the substrate actually is.
The pitch says the agent replaces the analyst. The evidence says the agent replaces the analyst's mechanical hours — 14 points of them in five months — and only where somebody already did the tedious work of writing down what the numbers mean. TextQL, Sigma and every other agentic BI entrant is selling into that same gate, and so is every platform vendor asking you to consolidate. None of them can define your metrics for you.
Grab's post says an analyst's impact moves from producing the artefact to owning the question and the call behind it — part builder, part advisor, part strategist. The narrower version its own data implies is the more useful one: the analyst decides what the number means, and that job just became the constraint on everything else.
Buy the definition, not the copilot. The copilot is the cheap part.
Continue Reading
- Tableau's 33M Semantic Models Take On Power BI Copilot
- Gartner: AI Winners Spend 4x More on Data, Not Models
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Airbnb Shipped 80% More Features. One Number Is Auditable.
- Governed Autonomy: Why Only 21% of Enterprises Got It Right
- Nvidia's $400M Kumo Bet: LLMs Can't Touch Your Database
- Snowflake Cortex vs Databricks Mosaic AI: Pick on Exit Cost
