Connecting an AI agent to your data over MCP does not make its answers right. The tools behind the protocol do. UNICEF's chief statistician, João Pedro Azevedo, ran a benchmark on UNICEF's own statistics. A generic MCP server scored 0.074 at returning the right figure for the right year. The same model with no tools scored 0.147. A purpose-built server on the same data scored 0.990. If your platform team is putting a generic connector in front of your warehouse, it is building the configuration that lost.
This matters this week because the UN has just made the same bet at global scale. On September 17 it opened nearly 44 million official data points from 26 UN entities through one gateway, and agents can query it directly. The benchmarks published around that launch are the most useful public evidence so far on a question every CDO is now answering: do you expose your metrics through a generic connector, or do you build tools that know which number is the right one?
What the UN Opened to AI Agents on September 17
The UN System Data Commons is a single gateway to official UN statistics that AI agents can query directly over the Model Context Protocol. It runs on Google's open-source Data Commons and supports MCP. Twenty-six UN entities have committed to it, and data from nearly 20 was live at launch. The target is 80% of the UN system's statistical datasets on the platform by 2027, and Google.org put in $2 million to build the core. MCP is an open standard that lets an AI application call outside tools and data sources through one common interface. Google shipped the Data Commons MCP server in September 2025. Its documentation lists six tools, from search_indicators to get_observations, and carries its own warning: "AI applications using the MCP server can make mistakes, so please double-check responses."
The case for it rests on a measurement. A UNICEF study, described by Azevedo, tested six models across more than 133,000 responses to questions about development indicators. The models were GPT-4o, GPT-4o-mini, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 2.5 Flash and Gemini 2.0 Flash. Average accuracy was 21.2%. About three in five responses gave no usable number at all, often because the model hedged. When the same questions were re-run about two days later on the same model versions, models that answered with a number both times gave the identical number only about half the time.
At the launch briefing, UNICEF Executive Director Catherine Russell described the failure mode plainly. The models, she said, "can produce extremely convincing answers while getting the facts wrong, confusing indicators, misinterpreting data, or failing to identify the authoritative source." That is not an academic worry. UNICEF estimates AI assistants now account for about one in 10 visits to its data website, and visits from people clicking links in ChatGPT answers rose 67% year over year.
Why a Generic MCP Server Scored Below No Tools
The generic server lost because the model had to chain four tool calls and then dig a number out of raw SDMX-JSON, and most of the time it could not. SDMX is the international standard official statistics agencies use to publish data. A generic connector is an MCP server that mirrors a data source's native API (search, describe, build a query key, fetch) and leaves the model to put those steps together.
Azevedo's benchmark is a small, clean design. There are three conditions: Claude Sonnet 4 alone, Claude with his purpose-built unicefstats-mcp, and Claude with sdmx-mcp, a generic SDMX server maintained under UNICEF's own GitHub organization. Every run was at temperature 0. The test covered 10 indicators across mortality, nutrition, maternal health and education, in two independent 20-country samples with no overlap; the generic server was run on the first sample only. Each sample had 300 queries: 100 asking for the latest figure, 100 asking for a named year, 50 asking for years with no data, and 50 asking about indicator-country pairs that have never been measured.
The score, EQA, multiplies three things: whether a number came back, whether it was for the right year, and how close it was to the official figure. If any one of them fails, the whole score falls.
| Condition | Accuracy (all queries, first sample) | Cost/query | Latency |
|---|---|---|---|
| No tools | 0.147 | $0.003 | 5.0s |
| Purpose-built MCP | 0.990 | $0.018 | 9.8s |
| Generic SDMX MCP | 0.074 | $0.087 | 60.0s |
The benchmark's own diagnosis of the generic server is one line: "raw SDMX-JSON is hard for the LLM to parse." The mechanics show what that means. The generic server needed an average of 3.7 tool rounds (search, describe, build_key, query_data) where the purpose-built one needed 2.0. The model pulled a number out of only 64% of generic-server responses, against 100% for the purpose-built one. Value accuracy averaged 0.11 against 1.00. Running 300 queries cost $26.20 through the generic server and $5.47 through the purpose-built one, and each answer took a full minute.
So the generic connector did not just fail to help. It spent nearly five times as much as the purpose-built tool to be less accurate than the model's memory.
What the Purpose-Built Server Did Differently
The winning server did the parsing before the model saw anything: it handed back values already extracted, in two tool rounds, where the generic one handed over raw SDMX-JSON. The version tested was v0.3.0, and it was simpler than today's. According to the changelog, the statistician's layers came after the benchmark: an indicator-name resolver in v0.7.0 in May, and the strict lookup_by_code tool in v1.0.0 later that month. Its README now lists nine narrow tools. lookup_by_code "rejects natural-language input" and accepts only a valid canonical code. get_temporal_coverage reports which years and countries actually have data before anyone asks for a value.
Two design layers, both added after the 0.990 run, define the server today:
- An indicator resolver. This layer turns "under-five mortality in Kenya" into the one canonical series code that answers it. It works by breaking the cryptic UNICEF codes into their meaningful parts: prefix families, methodology suffixes and age bands. In an enterprise, this is your certified metric catalog. It is the difference between
net_revenue_recognizedand the four other columns with "revenue" in their names. - Absent-data safety. In the current version, it layers strict lookup, ambiguity flags, confirmation requirements and advisory hints on the model's path to an answer. The server also caches the latest year each indicator has data for. When every fallback query comes back empty, it reports that as an authoritative "no data" answer rather than a tool failure the model might try to work around.
This costs something. Grounding raised per-query cost from $0.003 to $0.018, about 6x, and roughly doubled latency. That is the correct line item for a budget. The mistake is to assume the generic connector is the cheap option. At $0.087 per query and 60 seconds per answer, it was the most expensive configuration in the test and the least accurate.
Higher Accuracy Did Not Buy Safety on Missing Data
Grounding raised accuracy more than fivefold, but in the version tested it did not reduce fabrication on data that does not exist. Only the generic server got that to zero. This is the finding most write-ups of this launch will miss, and the one your evaluation most needs to copy.
On the questions about pairs that were never measured, Claude alone fabricated a figure 12% of the time on the first sample. With the purpose-built server, the raw rate was 34%. The generic server was at 0%. It was the safest configuration in the test, because its assistant_guidance fields and scope validation stopped the model from answering at all.
Much of that raw rate (34% and 40% on the two samples) was a scoring error. The ground-truth pipeline had marked mortality data for Andorra, Monaco, Palau and San Marino as never existing, when the SDMX API does hold UN estimates for them. After that correction the author puts the true rate at about 10%, which is roughly where the model was with no tools. For the remaining fabrications he offers an explanation he calls the "confidence effect." Where the model has strong prior knowledge, as it does for child mortality, it overrides the tool's "no data" answer and makes up a figure from its training. Where its prior knowledge is weak, it accepts the refusal.
The fix was a design change, not a model change. The changelog credits an absent-data safety layer added in v0.4.0 and the v0.7.0 indicator resolver. With v0.7.3 and its follow-up fixes, according to the project README, accuracy was 0.891 on a 40-country sample and hallucination was 1.00%, against 2.50% with no tools. The README calls it the first version where the MCP server made the model safer on absent data than using no tools at all.
The lesson for your own evaluation is that accuracy and safety on absent data are separate axes, and a single score hides the trade-off between them. We saw the same pattern when 12 models wrote a fake crate and none refused a real one: questions with no valid answer need their own test set. The same goes for a baseline. As the agent-memory benchmarks showed, if you never test with the feature switched off, you cannot tell when it is making things worse.
Read the Benchmark Before You Quote It
This is strong directional evidence, not proof, and its author says so in public. Here is the case against relying on it too heavily.
The same person built the winning server and designed the benchmark. It used one model, 10 indicators and 600 queries per condition (300 for the generic server). Azevedo's own design-issues file lists the benchmark's flaws. Two indicators account for 41.5% of all queries. Every never-existed test case comes from just eight countries. And 29 queries for one indicator were wrongly marked as absent because a totals filter hid disaggregated data the API actually had. It is unusual to publish a list like that, and it counts in the work's favor. It also means the 0.990 should be read as "near-ceiling on this indicator set," not as a general law. If you are about to rank tools on a few hundred questions, size your eval set first.
The generic server may also have moved on. The benchmark describes it returning SDMX-JSON. Its current README lists 27 tools, says data queries return CSV, and includes compact tools such as get_single_observation, get_indicator_table and resolve_and_query_data. Whether or not those existed when the benchmark ran, they make the same point from the other direction: a generic connector becomes useful by growing a purpose-built layer.
The new UN platform has the same weak point. An independent field guide to data.un.org, which says it is not endorsed by the UN or Google, reports a live test on September 18. "Child mortality in Bengal" silently returned US data, while "child mortality in Bangladesh" returned Bangladesh, and both charts carried identical titles. The guide calls this the resolver committing to a place without telling you. Neither the protocol nor the model caused that error. The resolver did.
Your Warehouse Is the SDMX Registry in This Story
Every enterprise data platform has the same two layers as UNICEF's: a query API, and the knowledge of which number is the right one. A generic connector exposes only the first. The list-tables, describe-schema, run-query server your team can stand up in an afternoon is the sdmx-mcp of this benchmark.
The enterprise evidence points the same way. A paired benchmark posted to arXiv in April by two researchers at Cube, which sells a semantic layer, tested three frontier models, including Claude Sonnet 4.6 and GPT-5.4, on 100 questions over a retail dataset in ClickHouse. Given only the warehouse schema, the models scored 45.5-50.5%. Given the schema plus a 4 KB hand-written document describing the measures, conventions and disambiguation rules, they scored 67.7-68.7%. With or without the document, the three models were statistically indistinguishable from one another. The document mattered. Among these three frontier models, the choice of model did not.
Vendors say the same thing about their own products. In 2024 Snowflake claimed 90%+ SQL accuracy for Cortex Analyst with a semantic model, against 51% for single-prompt GPT-4o. That result comes from its own 150-question internal set and should be weighed as a vendor's claim. The practitioner version is Grab's five months running analytics agents, where metric context, not model capability, was the ceiling.
Put the three together and the budget conversation changes. Model selection moves the number by a few points. What the tools hand the model, and the metric definitions behind them, move it by tens of points. It is also worth checking that the connectors you are counting on start at all before you argue about which one is smarter.
What to Do Before Agents Quote Your Numbers
Treat "expose it over MCP" as the start of the work, and fund the layer that decides what the right number is.
This Week:
- Inventory the generic connectors. List every assistant or agent that holds a warehouse, SQL or BI connector, and pull the last month of metric questions it was asked. That list is your test set's first draft.
- Write the golden set. Take 50 figures your executives actually quote, each with its official value and period. Add 10 questions whose honest answer is "no data": a metric that was never tracked for a region, or a quarter that has not closed.
This Month:
- Run three arms, not two. Test no tools, the generic connector, and a narrow purpose-built tool. Score four things separately: whether a number came back, the right period, the right value, and fabrication on the no-data questions. Re-run it two days later. UNICEF's six-model test found repeat answers matched only about half the time.
- Write the 4 KB document. Measure definitions, fiscal calendar, which table is canonical, and how to break ties between similar metrics. The arXiv result says this alone is worth roughly 20 points.
- Build a strict lookup. Make one tool that accepts only a certified metric ID and rejects free text, and have it return "no data" as an answer rather than an error. UNICEF's server first got hallucination below the no-tools baseline with a resolver and an absent-data safety layer, and added the strict lookup after that.
Before You Fund the Platform:
- Budget for grounding honestly. Plan on roughly 6x the per-query cost of an ungrounded model and about twice the latency. If your chat interface is designed around 5-second answers, a generic connector's 60 seconds is not a tuning problem.
- Stop letting assistants quote macro figures from memory. FP&A, strategy and risk teams citing country or development statistics should point agents at the UN's MCP endpoint, and keep a human on the output. Google's Data Commons lead, Prem Ramaswami, says "a human should always review the outputs before citing or publishing them."
The Bottom Line
MCP solves the connection problem and leaves the meaning problem exactly where it was. This has happened before. Standard database drivers gave every reporting tool a way to connect to every database, and finance teams still argued over whose revenue number was right until metric layers and certified definitions caught up. MCP is that connection standard for agents. It is useful, it is necessary, and it is not the thing that makes the answer right.
UNICEF's chief statistician built his own server by hand, tested it on 10 indicators, and got near-perfect accuracy. He also published the flaws in his own benchmark, and he found that his better tools did not, at first, make the model any less willing to make things up. That is what honest grounding work looks like, and nobody gets it by installing a connector.
The protocol gets the agent to the data. Only the resolver gets it to the right number.
Continue Reading
- Grab Ran Analytics Agents 5 Months. Context Was the Ceiling.
- 150 MCP Servers Never Started. No Benchmark Shows That.
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- 12 Models Wrote a Fake Crate. None Refused a Real One.
- Agent Memory Cost 14 Points at Best. Test With It Off.
- The 2026 Agentic AI Stack: 8 Layers, 3 You Can Skip