Your agent platform shortlist almost certainly has a tool-calling benchmark score in it. That score was earned on a corpus where the same handful of tool definitions recur across task after task, and it says nothing at all about the failure that will actually take your agent fleet down.
A preprint posted to arXiv on 10 September 2026 drew 400 servers at random from a 24,135-server census of the Model Context Protocol registry and probed every one over the wire. Only 48.8% completed an initialize handshake. The dominant failure was not a missing API key. It was servers that simply never start — 150 of the 400, or 37.5%, nearly three times the 13.3% blocked by missing credentials. It is single-authored and not yet peer-reviewed, which matters less here than it usually would: the seed, the probe pipeline and the per-server outcomes are all published under a Zenodo DOI, so the draw regenerates and anyone can check it. The larger ecosystem studies before it either curated their sample or repaired servers until they ran, so each one erased exactly this quantity.
What 400 Randomly Drawn MCP Servers Actually Did
A probability sample is a draw where every member of the population has a known chance of selection — no popularity list, no reference set, no repair step. The author took a census snapshot dated 2026-08-22, filtered to npm-published servers declaring stdio transport, sorted canonically by registry identifier and drew 400 with a published seed of 20260819 and a partial Fisher-Yates shuffle, so the draw regenerates. An initialize handshake is the first JSON-RPC exchange in MCP: the client says hello, the server declares its capabilities, and only then can tools be listed.
| Outcome | Servers | Share |
|---|---|---|
| Completed the handshake | 195 | 48.8% |
| Never started at all | 150 | 37.5% |
| Needed credentials | 53 | 13.3% |
| Package unavailable | 2 | 0.5% |
Measured with the same instrument, a hand-curated frame of 24 reference and popular community servers came in at 66.7% — a gap of 17.9 percentage points that exists purely because somebody chose the list. And the author is explicit that 48.8% is a lower bound: inclusion was decided by a single probe with no retry, so transient failures count against it.
The population is also moving fast. Between the 2026-07-14 and 2026-08-22 snapshots the registry went from 16,548 to 24,135 servers, about 195 net new servers a day, with remote-only entries growing 70.1% while the npm/stdio slice fell from 35.1% to 30.7% of the population. That is the paper's own scope limit, and it is the right one to hold in mind: this is a clean measurement of a third of the ecosystem, not all of it. The remote half measures considerably better. A July 2026 census by Health AI — which sells MCP verification, and so has every commercial incentive to report breakage — probed 9,326 remote servers and found 82.8% reachable. So 48.8% is not an ecosystem-wide availability figure, and the transport growing fastest is the healthier one.
Why Every Earlier MCP Study Looked Healthier Than This
Earlier ecosystem studies were not wrong; they were answering a different question, and their method deletes exactly the number an enterprise needs. The paper names three. One tracks 120 snapshots across 19,099 servers for metadata drift. MCPZoo, the largest runtime collection so far, runs a repair pipeline that converts 64,611 collected servers into 37,288 that support dynamic analysis. A third dynamically audits 414 internet-facing servers for vulnerabilities. As the author puts it: "Each is a reasonable engineering answer, and each erases the same quantity: how much of the published population is simply dead on arrival."
If you are building a research corpus, repairing broken servers is correct — you want signal, not noise. If you are an architect planning an agent fleet against a registry your developers will point at, the repair step deletes your entire operational risk. This is the same gap between a vendor's curated demo catalogue and a live registry that we covered when Snowflake bought Natoma to put a trust layer in front of 9,400 MCP servers. The official registry is run as a community metadata catalogue, with namespace ownership verified through GitHub OIDC or DNS. Publisher identity is checked. Whether the thing runs is not.
The Benchmark Measures a Problem Your Registry Doesn't Have
Tool-use benchmarks score a model's ability to pick the right tool from a set of confusable ones — and real MCP tools are barely confusable at all. Across the 2,766 tools advertised by the 195 live servers, only 10 were exact name-plus-description repeats: 0.4%. Run the same measurement over the tool definitions these benchmarks attach to their tasks and 68.8% of BFCL v4's and 85.6% of UltraTool EN's are exact repeats — 6,002 of 8,726 and 12,052 of 14,084 extracted definitions. Be precise about what that is: it counts tool definitions, not test cases, and a benchmark that reuses one tool across many tasks produces a high rate by construction. The paper's own conclusion is correspondingly narrow — any statistic computed over these files without global deduplication "measures how often a benchmark repeats a task, not how many tools it contains."
Near-duplication tells the sharper story. At a TF-IDF cosine threshold of 0.70, real MCP tools show 2.8% near-duplication and all of it sits inside single servers — cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. BFCL was built from "expert curated, and user-contributed functions and associated prompts" and published at ICML 2025; nothing about it is dishonest, but a corpus assembled that way accumulates the same function under different task framings.
Steel-man the other side properly, because the paper does. UltraTool, an ACL 2024 Findings benchmark built from real-world complex queries, shows just 0.3% near-duplication after global deduplication — cleaner than real MCP tools. So this is a property of BFCL specifically, not an indictment of synthetic corpora as a class. And 2.8% is not zero: two tools on the same server genuinely do collide, so intra-server disambiguation is a real, if small, engineering problem.
What does not survive is the inference procurement teams keep making. A high tool-selection score demonstrates a model can disambiguate near-identical tools presented side by side. In your registry, tools from different authors do not look alike. The score is evidence of a skill your workload will rarely exercise, and it is silent on whether the server answers. It is the same class of error as reading a single benchmark round as a model ranking, or sizing an eval set too small to distinguish two models at all.
An independent validity audit of tool-calling evaluation published in June 2026 found an 18.5% misalignment rate across 496 expert-reviewed tasks spanning BFCL v4, τ²-Bench Retail, LiveMCPBench and MCP-Atlas, and an 18.9-point score spread on LiveMCPBench across 23 reruns of the same pipeline. Two independent teams, two different attacks — the corpus and the scorer — arriving at the same place.
Half of Real Tools Won't Tell an Agent If They Delete Things
Hard protocol conformance among live servers is perfect; the optional safety metadata is where the variance lives, and a random draw is materially worse than a curated one. Zero of 2,766 tools carried a fatal JSON Schema violation and zero of 195 servers had any — no missing schema, no invalid type, no malformed properties or required. That is a genuinely good result and worth saying plainly.
Then the annotations. MCP defines four optional hints — readOnlyHint, destructiveHint, idempotentHint and openWorldHint — which tell an agent whether a tool is safe to call before it calls it. On the random draw, 1,626 of 2,766 tools (58.8%) carry none, against 41.5% on the curated frame: another 17.3-point curation flattery. Servers are all-or-nothing about it — of 194, 122 annotate no tools and 72 annotate every tool, with a one-sided 95% upper bound of 1.53% on partial annotation.
Do not build a control on this field regardless of coverage. The specification is unambiguous: "clients MUST consider tool annotations to be untrusted unless they come from trusted servers." A hint is a claim by the server author about their own code. Treat destructiveHint: false as documentation, and put the actual enforcement in your gateway and your agent's identity — which is the argument behind issuing agents real identities rather than inferring intent, and why an allowlist that trusts a declared command name is not an allowlist.
The Number That Quietly Sizes Your Context Budget
Tool counts per server are skewed hard, and four average servers put you at the edge of where model tool-selection starts degrading. The sample found a minimum of 1 tool per server, a median of 8, a 95th percentile of 46 and a maximum of 300. Anthropic's own documentation says tool-selection accuracy degrades once you exceed 30-50 available tools, and that a typical five-server setup — GitHub, Slack, Sentry, Grafana, Splunk — consumes roughly 55k tokens in definitions before the model does any work.
Do the arithmetic against the measured distribution. Four median servers is 32 tools: you are already in the degradation band. One 95th-percentile server is 46 on its own. A single 300-tool server exceeds it six times over. That is why deferred loading and tool search exist — the documented mitigation cuts definition context by over 85% by loading only the 3-5 tools needed — and why Claude and comparable platforms now cap deferred catalogues in the thousands rather than assuming everything fits. The point for an architect is that the constraint is measurable from your own registry today, and it has nothing to do with any published score.
What To Do Before the Next Vendor Review
This Week:
- Pull your own registry — internal catalogue, gateway allowlist, whatever your agents actually resolve against — and run a
tools/listprobe on every entry. You are looking for one number: what share completes an initialize handshake on the first attempt, no retries, no repairs. Anything materially below your assumption is your real availability ceiling. - Count tools per server and sum the total exposed to any single agent. If it is over 30, tool selection is already being tested outside the range the vendor benchmarked.
- Grep your tool inventory for
destructiveHintandreadOnlyHintcoverage. Expect roughly half to be absent, and confirm no policy anywhere depends on them.
This Month:
- Replace the benchmark score in your evaluation matrix with a corpus drawn from your own tools. Deduplicate it globally before you compute anything — on public corpora that step removed 69% to 86% of extracted tool definitions, and there is no reason to assume an internally assembled set is cleaner.
- Add a start-rate SLO to your MCP gateway alongside latency and error rate. Dead-on-arrival servers fail silently at handshake, not at call time, so they show up as an agent that "didn't find the tool" rather than an outage.
- Re-draw and re-probe monthly. At ~195 net new servers a day, and with the remote transport share growing 70.1% between two snapshots six weeks apart, a one-off audit is stale before the quarter closes.
Before You Sign:
- Require the vendor to state, in the contract or the architecture document, which corpus their quoted tool-use figure was computed on and whether it was globally deduplicated. A vendor who cannot answer is quoting a number they did not verify either.
- Ask what happens to your agent when a registered server fails to start mid-session — retry, fail open, fail closed, or hang. Most teams find out in production.
The Bottom Line
We have run this cycle before. Public API directories in the 2010s counted listings, not working endpoints, and the gap between "registered" and "responds" was where every integration project lost its schedule. Package registries taught the same lesson with more expensive consequences. MCP is now at 24,135 registered servers and growing at nearly 200 a day, with a protocol whose hard conformance is genuinely excellent and an operational floor nobody had measured until this week. The layers you assemble above it cannot fix a server that never answers.
The uncomfortable part is not that the benchmarks are wrong. It is that they are answering a question your registry never asks, while the question it does ask — does this thing start? — has no leaderboard at all.
A score you did not compute on your own corpus is someone else's measurement of someone else's problem. Go probe your registry.
Continue Reading
- 9,400 MCP Servers, 18% Trust: Why Snowflake Bought Natoma
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- DeepSeek Won Solo. Gemini Won the Arena. Run Both Rounds.
- Your Agent Allowlist Says Python. It Means Any Command.
- 12 Models Wrote a Fake Crate. None Refused a Real One.
- Okta vs Entra Agent ID vs SailPoint: Two Issue, One Governs
- The 2026 Agentic AI Stack: 8 Layers, 3 You Can Skip
- MCP Goes Stateless: Enterprise AI Governance Just Got Real
