Before you let an AI agent buy for a customer, Visa's first benchmark says to plan for the agent being wrong about one time in four, and for it to spend the money anyway. Across 22 models from 12 labs, Visa's November 5 release reports a 57.2% pass rate on 15 realistic shopping tasks. Only 34.2% of model and scenario pairs passed on all three attempts, and about 54% of the failed runs still ended in a purchase. The practical response is three changes: test agents on whether they pass every repeated run, score them on whether they ask the right questions before buying, and keep a human confirmation step in front of the payment.
The benchmark measures product discovery, the conversation before checkout. Visa says checkout, payment and order placement come in later rounds. So these numbers describe the step a merchant or bank can least afford to get wrong quietly: choosing what to buy.
What Visa's Agentic Commerce Benchmark Actually Tested
The Visa Agentic Commerce Benchmark is a set of 15 simulated shopping conversations in which an agent has to work out what a customer needs and then either buy the right product or decline to buy. Tasks include buying a TV, a laptop and a personal care appliance. Each model ran each scenario three times.
The design choice that matters is that the simulated customer does not state every requirement up front. In the TV scenario, the shopper mentions a brand a relative recommended and a membership with a third-party supplier, and the agent has to draw out the budget, format compatibility, HDR10+ support for family videos and gaming features through conversation. A run passes only if the agent discovers every requirement the task defines and makes the correct purchase or no-purchase decision. When no product fits, the right answer is to not buy, and the agent has to gather enough information to justify that call, which rules out a lucky guess.
That is closer to a real customer than most agent benchmarks get. People rarely type a full spec into a chat window. They say "something for the living room, my kids game a lot," and the agent either asks or assumes.
How Often Did the Shopping Agents Fail?
The agents failed 42.8% of the time overall, and the spread between models was wide. Per Visa's release:
| Measure | Result |
|---|---|
| Overall pass rate | 57.2% |
| Best model | 82.2% |
| Worst model | 28.9% |
| Passed all three attempts | 34.2% of cases |
| Failed runs that still made a purchase | about 54% |
| Top five at uncovering hidden requirements | 70.2% pass |
| Bottom five at uncovering hidden requirements | 44.4% pass |
| Five most expensive models per run | 67.1% pass |
| Five least expensive models per run | 50.2% pass |
Multiply the two headline failure numbers and roughly 23 of every 100 runs (42.8% times 54%) ended with the agent buying something in a run it had failed. Visa names two ways that happened: recommending a product that missed the customer's stated needs, and buying before it had found all of them.
Visa did not name the models. TNW, citing Fortune's reporting, put agents from Google and Meta tied at the top with 82%, a Claude agent third at 76% and an OpenAI agent sixth at 64%. Treat that ranking as secondhand. It also deserves less weight than it will get in vendor decks: with 15 scenarios and three attempts, each model ran 45 times, so 82.2% is 37 passes and 76% is about 34. Three runs separate first place from third. We covered why gaps that small rarely survive a bigger test set in Only 3 of 36 Model Gaps Were Real.
Why the 34.2% Consistency Number Matters More Than the Pass Rate
Repeat consistency is the share of tasks an agent passes on every attempt, and it tells you which tasks the agent handles reliably. A single test run tells you an agent can do a task. Your customers get every run, including the bad ones.
The gap here is 23 points: 57.2% of individual runs passed, but only 34.2% of model and scenario pairs passed all three times. Visa adds that models which uncovered more requirements passed more often but often failed when they repeated a scenario they had already passed. A vendor demo, or an internal pilot that ran each test case once, measures the 57%. The share of tasks you can count on the agent to get right for every customer who asks is nearer the 34%.
The industry has seen this result before. In 2024, the τ-bench paper from Sierra researchers introduced pass^k, the probability that an agent succeeds on all k attempts at the same task, and reported that function-calling agents including GPT-4o succeeded on fewer than half of tasks, with pass^8 below 25% in its retail domain. Two years and several model generations later, a payments network running its own shopping test finds the same shape: higher single-run scores, and a large drop once every attempt has to pass.
The strongest counterargument comes from Visa itself. Its FAQ says the results do not mean agents are unready for commerce, only that consistency and reliability remain problems, and the best models passed more than four runs in five. That is fair. But a merchant does not get to choose which run a customer lands on, and an 82% agent that fails nearly one customer in five is a returns and chargeback problem at any real volume.
What the Data Says to Buy: Elicitation, Then Spend
The capability that separated good agents from bad ones was asking questions. The five models best at uncovering hidden requirements passed 70.2% of runs against 44.4% for the bottom five, a wider gap than the one between the most and least expensive models (67.1% against 50.2%).
Price still correlates with quality, so the cheapest model is unlikely to be the right one for a customer-facing purchase. But if you are choosing between models for a shopping or procurement agent, test how each one handles an underspecified request before you look at per-token cost. An agent that asks "is this for gaming?" before buying a TV will cost less in returns than the token savings from a cheaper model.
Visa's own consumer data shows how little room there is for error. Its September Trust Index found that 72% of US consumers had used an AI assistant but only 23% trusted AI to handle payments for them. Rubail Birwadker, Visa's global head of growth, told Fortune, per TNW: "We're not there yet from a trust or technology perspective."
Banks Are Already Setting Agent Policy
Financial institutions are restricting agents before the reliability problem is solved, and Visa's numbers give them evidence to point to. Per Dealroom's summary of a November 8 Telegraph report, HSBC, Barclays, Lloyds, NatWest and Nationwide are blocking or restricting AI agents that can transact on a customer's behalf, while leaving room for controlled, read-only access through approved open-banking services. Dealroom notes the curbs look precautionary, with no reported cases of agents getting into UK accounts. The same report notes JPMorgan, Citi, Charles Schwab and Wells Fargo limited agents using customer credentials in October, which we covered in Schwab and JPMorgan Block AI Agents Using Customer Logins.
Regulators are moving in the same direction. On October 19, the FCA's Jessica Rusu said the regulator will work with industry and frontier labs on a framework for agentic finance with "clear approaches to agent identity, delegated authority, accountability and execution". In the same speech, reported by PYMNTS, she said agentic systems "require continuous, conditional assurance."
Identity is only part of the answer, and the benchmark makes that plain. Visa's Trusted Agent Protocol lets a merchant verify an RFC 9421 message signature to confirm a request comes from a Visa-approved agent instead of a bot. A valid signature tells you who the agent is, and nothing about whether this run understood that the customer needed HDR10+. A fully verified agent could still produce any of the failed runs in Visa's test that ended in a purchase. We covered the protocol stack in Visa's 4-Protocol Bet and the competing Personal Agent Protocol.
What to Do Before You Open Checkout to Agents
This Week:
- Re-run your agent evals with repeats. Take whatever test set you used to approve a shopping, procurement or support agent and run every case at least three times. Report the share that passes every attempt next to the average. If the two numbers are 20 points apart, as Visa's are, your launch decision was based on the wrong one.
- Add underspecified cases. Write five test conversations where the customer leaves out a requirement that should change the answer, plus at least one where the correct outcome is "do not buy." Score the agent on whether it asks before it acts. Tools such as Braintrust and LangSmith can run repeated trials and store the transcripts.
This Month:
- Put a confirmation step in front of every payment. The agent proposes the item, the price and the reasons it fits the stated needs; the customer approves. Visa has not yet tested the payment step at all, so nothing in this data supports letting an agent complete a purchase unattended.
- Write your agent-access policy if you are a bank or merchant. Decide which actions third-party agents may take (browse, build a cart, pay), how they identify themselves, and who carries a disputed purchase. The UK banks drew the line at transactions; Uber's experience shows how little volume third-party agents bring today, so a restrictive default costs you little.
Before You Pick a Model:
- Ignore leaderboard gaps of a few points. At 45 runs per model, first and third place in Visa's test were three runs apart. Run your own scenarios at a sample size that can separate the candidates, and weight elicitation and consistency above price.
The Bottom Line
Agentic commerce is repeating the early web payments pattern: the protocols and identity layers are arriving first, and the question of whether the buyer's instructions were carried out correctly is left for later. Card networks eventually answered that one with chargeback rules, which put the cost of a disputed transaction on whoever could best prevent it. Visa's benchmark is a first look at how often an agent will generate that dispute. Visa says the next rounds will add product categories and extend testing to checkout, payment and order placement.
Run your own evals three times each before that round comes out, and compare the all-three pass rate with Visa's 34.2%.
Continue Reading
- Schwab and JPMorgan Block AI Agents Using Customer Logins
- Third-Party Agents Brought Uber Four Rides in a 320M-Trip Week
- Meta's Personal Agent Protocol Signs Walmart Before It Has a Spec
- Amazon Blocks Meta's Muse Agent Where the Perplexity Ruling Stops
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.