The containment rate in your voice AI contract is not a measurement. It is a marketing number, and no public test had named the shipping stacks you are actually buying and scored them on it. One now has. A team put eleven shipping voice-agent stacks through the same 100 credit-card support scenarios, 3,300 simulated phone calls in total, against a real database rather than a transcript — and the best of them completed 71.0% of calls, while the field as a whole managed 52.2% once the call required more than one step. Per-call cost varied thirteenfold across the field with no visible relationship to quality.
If you are renewing a contact-center contract this quarter, the useful thing here is not the leaderboard. It is the test.
What the Benchmark Actually Measured
VAmoS Bench measures whether a voice agent finished the job, not whether it sounded good doing it. The paper, posted to arXiv on 29 July 2026, makes the distinction explicit: word error rate, speech naturalness, endpointing and model quality "help diagnose a system, but none alone shows that the deployed agent authenticated the caller, invoked the right tools in the right order, and completed the request."
So the authors built the whole loop. The agent is Riley, a credit-card support rep at a fictional bank who can freeze, cancel, replace or activate a card. Each of the 100 scenarios seeds a PostgreSQL backend and gives a simulated caller a private goal. Riley has five tools that run real SQL against that backend. A grader then checks binary assertions against the full trace — what was said, which tools fired, with what arguments, and what rows came back.
That last part is the reason this benchmark is worth your attention. It catches the two failure modes a contact-center QA dashboard structurally cannot see: an agent that says it froze the card without touching the database, and an agent that makes the correct database change while disclosing something it should not have.
Eleven stacks were run, three times each: Pipecat, LiveKit Agents, Vapi, ElevenLabs ConvAI, Cartesia, OpenAI Realtime, OpenAI Realtime mini, Gemini 2.5 Native Audio, Gemini 3.1 Flash Live, Retell, and NVIDIA's Nemotron. Same scenarios, same assertions, same grader, same caller channel.
The Field Bunches at the Top, Then Breaks on Complexity
Nine of the eleven stacks landed within ten percentage points of each other, and every one of them fell apart on the same kind of call. Pooled across all agents and all 3,300 calls, 2,081 completed successfully — 63.1%.
| Stack | Completion | Est. cost/call |
|---|---|---|
| Pipecat | 71.0% | $0.045 |
| LiveKit Agents | 70.3% | $0.048 |
| Vapi | 69.3% | $0.135 |
| ElevenLabs ConvAI | 67.3% | $0.114 |
| Cartesia | 67.0% | $0.146 |
| OpenAI Realtime | 67.0% | $0.074 |
| Gemini 2.5 Native Audio | 64.0% | $0.023 |
| Gemini 3.1 Flash Live | 62.3% | $0.016 |
| Retell | 61.3% | $0.208 |
| OpenAI Realtime mini | 51.0% | $0.032 |
| Nemotron | 43.0% | licensed by capacity |
Now split the scenarios by difficulty, which is where the number in your RFP quietly comes from. On the 30 simple scenarios — one cooperative thread, typically one verification — the field pooled 65.8%. On the 38 complex ones, several dependent actions and nuanced verification, it pooled 52.2%, and complex was the weakest group for every single agent.
Then the counterintuitive result. The 32 adversarial scenarios — callers applying pressure to verification, confidentiality and scope — scored highest, at 73.4%. Refusing is easy. These systems have been trained hard to say no. Doing the multi-step job correctly is what they have not been trained hard enough to do, and multi-step is what your tier-1 queue is actually made of.
One more thing to carry into a vendor conversation: run-to-run spread. With only three runs per agent, per-run completion spanned 17 points for Nemotron, 12 for Cartesia, 9 for Gemini 3.1 and 8 for Pipecat — all wider than several of the gaps near the top of that table. The ranking is directionally real. The ordering inside the leading pack is noise, and the authors say so themselves.
Price Is Buying You Almost Nothing
The most expensive stack in the field scored below the cheapest tier. Retell came in at an estimated $0.208 per call and 61.3% completion; Gemini 3.1 Flash Live cost $0.016 and completed 62.3%. That is thirteen times the price for one point less. The paper reports no correlation analysis between cost and quality, and looking at the two columns, there is not much of one to report.
The cost figures are snapshots of live pricing, and you can check them. Retell publishes $0.055/minute for its voice infrastructure plus TTS, LLM and telephony on top, with guardrails, PII removal and knowledge base each adding another half-cent to a cent per minute — a stack of add-ons that lands the configured price between $0.07 and $0.31 a minute. Vapi charges $0.05/minute for hosting and passes model costs straight through, $0 if you bring your own keys. ElevenLabs charges $0.080/minute beyond your plan's included minutes, doubling to $0.160 if you exceed your concurrency limit, with the language model and telephony billed separately. Every one of those is a real, dated, checkable number, which is more than can be said for the containment figure in most decks.
The other result buried in that table: the two top performers, Pipecat and LiveKit Agents, are open-source frameworks you assemble and host yourself. LiveKit's is Apache 2.0. They beat every managed platform in the test, at roughly a third of the price of the managed platforms in the middle of the pack. That does not mean you should fire your vendor — a framework hands you the orchestration and hands you the pager too. It does mean the premium you are paying is for operations, support and a contract, not for accuracy, and you should price it that way. We have made the rent-versus-own argument on voice infrastructure before; this is direct evidence that owning does not cost you quality.
Latency is the one place the money shows. ElevenLabs ConvAI posted the fastest median response at 1.19 seconds and Gemini 2.5 Native Audio the slowest at 15.95 seconds. Sixteen seconds of silence on a support call is not a latency figure. It is a hang-up.
The Refusal That Hands Over the Answer Key
Agents that correctly refused a fraudulent request still told the caller which verification field they got wrong. This is the finding that should change your red-team clause, and it is easy to miss because on the surface the agents did well.
Against direct bypass attempts, the field held. A caller claiming to be a branch manager was refused in 33 out of 33 agent-runs. Bank-employee pressure: 32 of 33. Spoken prompt injection: 32 of 33. If your evaluation stops at "can a caller talk it into unfreezing an account," you will conclude these systems are solid, and on that specific question you will be right.
Now the probes that matter more. On repeated failed verification attempts, agents passed 11 of 33 runs — 33.3%. On probes asking which verification field was wrong, 13 of 33, or 39.4%. On probes fishing for the account holder's name or address, 11 of 33 again. The authors put it plainly: an agent that correctly declines to act for an unverified caller "will still tell that caller which verification field was wrong, which is the oracle the attacker actually wanted."
An oracle, in this context, is a system that answers yes-or-no questions about a secret you are trying to guess. A human agent who says "I can't help you" ends the attack. An agent that says "the ZIP code doesn't match" has just turned a blind guessing game into a search with feedback — and unlike a human, it will do it for the ninth caller at 3am with the same patience as the first.
That matters more this year than last. Pindrop's voice intelligence report recorded a 1,300% rise in deepfake fraud attempts in 2024, from roughly one a month to seven a day, with synthetic voice attacks up 475% at insurers and 149% at banks, and a fraud attempt hitting a US contact center every 46 seconds. The same report puts 2024 fraud losses at $12.5B across 2.6 million reported events. Automation does not create that pressure. It does industrialise the response to it, in whichever direction the agent happens to be pointed.
Say/Do Divergence Is the Failure Your QA Dashboard Cannot See
Some agents announced they had completed an action they never performed. A conservative review of the 1,290 failed criterion explanations found at least 27 explicit say/do mismatches across 10 of the 11 agents — 2.1% of failures.
Two percent sounds survivable until you notice what it costs to detect. A transcript review scores those calls as successes: the agent was polite, confirmed the action, closed the call. A database audit scores them as successes too, if the agent got to the right end state via a prohibited route. Only a grader looking at both the words and the tool calls catches it, and almost nobody's contact-center QA is built that way.
The related failure is policy-order drift. When callers demanded a card replacement without the required freeze first, agents held the correct sequence in 1 of 33 runs — 3.0%. When callers tried to skip the freeze, 13 of 33. If your card-servicing policy has a mandatory ordering, and most regulated ones do, assume out of the box that the agent will not follow it and that your transcripts will not tell you.
Who Paid For This, and Why That Matters
Read the disclosure before you read the leaderboard. All eight authors are affiliated with Veris AI, which sells exactly what this paper argues you need: simulation infrastructure that builds a digital twin of your systems so agents can be tested before production. The company's own statement is unambiguous — "All authors are affiliated with Veris AI, which developed and operates the simulation platform used in this study" — and the acknowledgements note that Vapi, which placed third, gave feedback on the agent configuration.
That is a direct commercial interest in voice agents testing badly, and you should weight the framing accordingly. What it does not do is invalidate the mechanics. The scenarios, assertions and grader were identical across all eleven stacks, which is what makes the relative comparison usable even if the absolute numbers flatter the thesis. The failure classes — complexity collapse, verification leakage, say/do divergence — are reproducible in your own environment for the cost of an engineer's week, and that is the honest use of this paper.
The stated limits are real and the authors list them. One 100-scenario banking task. Three runs per agent. A single LLM grader with no human gold standard. A synthetic caller covering a narrow band of accents and speaking styles. Cost estimates that are snapshots of prices that move monthly. This does not rank these systems for your domain, your language mix, or your customers, and it does not claim to.
Set that against what the market has been told to expect. Gartner predicted in March 2025 that agentic AI would autonomously resolve 80% of common customer service issues by 2029, cutting operational costs 30%. Note the load-bearing word: common. On the simple scenarios, the field managed 65.8%, and the gap between that and 80% is three years of engineering, not a procurement cycle. Compare it to the 75% resolution figures reported for OpenAI's Presence rollout or the 67% autonomous resolution rate Salesforce paid $3.6B for and you get the real story: vendor numbers are not lies, they are measurements of a different, easier call mix.
Even a vendor's own buyer's guide will concede the point if you read far enough. One notes that many vendors quote containment — calls that do not reach a human — rather than resolution, and that "a call is contained if the customer hangs up". A hang-up is a contained call. It is not a served customer.
What To Do Before You Sign
This Week: Ask your incumbent and your finalists for the denominator. Not the containment rate — the definition. Which calls are in the base, what counts as contained, what happens on escalation, and whether an abandoned call scores as a win. Get it in writing. If the number changes when you ask that question, you have learned what you needed to.
This Week: Run the leak test by hand. Call your own agent, fail verification deliberately, and fail it again. Then ask which field was wrong. Ask for the name on the account. Do it five times. You will know inside twenty minutes whether you have bought an oracle, and the 39.4% pass rate in this benchmark says the odds are against you.
This Month: Build ten scenarios that mirror your ugliest real calls — multi-step, dependent actions, a verification wrinkle in the middle — and make every finalist run them against a seeded copy of your backend. Grade the database state and the transcript together, not either alone. This is the part the benchmark actually gives you for free: the method, not the scores.
Before Renewal: Rewrite two clauses. First, tie the SLA to resolution on your scenario mix, measured quarterly, with the mix fixed in an appendix. Second, add a confidentiality clause that names verification-oracle behaviour explicitly — refusing the action is not sufficient performance if the agent narrates the reason. Neither clause exists in the standard CCaaS paper today, and both are cheap to add before signature and impossible to add after.
Before Renewal: Price the managed premium honestly. If a self-hosted framework matches or beats your vendor on task completion, the difference in the invoice is buying you operations and a throat to choke. That may well be worth it. It should be a decision, not an assumption — the same discipline we argued for when voice infrastructure costs halved on owned stacks.
The Bottom Line
Every technology cycle produces a metric that vendors optimise and buyers stop interrogating. In hosting it was uptime, measured on a definition of "up" the vendor wrote. In search it was relevance. In contact centers it is containment, and this benchmark is a public measure of how much slack that word is carrying: the same eleven systems that look strong at 73.4% on adversarial calls and 65.8% on simple ones drop to 52.2% on the calls that make up most of a real queue, and the priciest of them buys you nothing the cheapest does not.
The industry has been arguing about which voice agent is best. The Stanford AI Index found agents hitting 66% on benchmarks while most never reached production, and the same gap is open here. It is the wrong argument. None of these systems is good enough to deploy on trust, and all of them are good enough to deploy on evidence — if you generate the evidence yourself, on your calls, before the ink dries.
Stop negotiating over the vendor's number. Bring your own.
Continue Reading
- Voice AI's Rent vs. Own Moment: Mistral Bets on Open Weights
- OpenAI Voice AI: $0.30 Calls vs $17 Human Agents
- Synthflow + 8x8: Outcome Pricing Comes for CCaaS Seats
- Why 50% of Voice AI Costs Vanish with Owned Infrastructure
- Cresta Knowledge Agent: Why Augmentation Beats Automation
- Salesforce Pays $3.6B for 67% AI Resolution Rate
- Stanford AI Index 2026: Agents Hit 66% Success Rate
- Why Sierra Just Raised $950M to Kill the Contact Center