A bake-off with no opponents cannot tell you which model wins a market. If the agent you are about to buy will set prices, submit bids, or negotiate terms against counterparties who are themselves running agents, then every number in your evaluation was produced in an ecology that does not exist in your business.
That is not a thought experiment any more. ERPBench, submitted to arXiv on 4 September 2026 by a team at GAIR-NLP, ran the same 100 enterprise-planning problems through six model families twice — once against fixed rule-based opponents, once with all six models competing in one shared market. The leaderboard inverted. DeepSeek led the first setting; Gemini led the second. The two settings picked the same winner on 21 of the 100 problems.
The Same 100 Problems, Two Different Winners
The finding is a ranking instability, not a scoring quibble: change who the agent is competing against, hold everything else fixed, and the model you would have selected changes.
ERPBench is an execution-instrumented simulation of an ERP business — six decision rounds of 30 simulated days each, in which the agent sets prices, production quantities, procurement, marketing allocation and financial actions for 12 SKUs across three distribution channels, under batch production limits, lot sizes, supplier lead times, cash-flow and credit dynamics, carbon costs and a competitive demand model. Score is terminal company valuation from a dividend discount model.
The same 100 seeded problems were run in two matched ecologies. In Solo, each candidate model competes against five fixed rule-based opponents — conservative, aggressive, balanced, growth-oriented and risk-averse policies. In Arena, all six candidate models compete in a single shared market. Six families were tested: Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro and Qwen 3.7 Max. That is 1,200 model-level trajectories across 7,200 decision rounds.
| Model | Solo valuation | Solo rank | Arena valuation | Arena rank | Δ |
|---|---|---|---|---|---|
| DeepSeek | 252.29M | 1.67 | 234.71M | 2.36 | −17.58M |
| Gemini | 169.41M | 3.51 | 263.95M | 1.76 | +94.54M |
| Qwen | 205.87M | 3.27 | 187.13M | 4.01 | −18.74M |
| GPT-5.5 | 186.43M | 3.87 | 194.82M | 3.91 | +8.39M |
| Doubao | 181.27M | 4.10 | 188.17M | 4.28 | +6.90M |
| Claude | 176.78M | 4.58 | 173.95M | 4.68 | −2.83M |
Gemini is the case that should worry anyone who signs off a model choice. In Solo it is a mid-table result — a 3.51 mean rank and a 22% bottom-rank rate. In Arena its bottom-rank rate is 0% and it leads the field. A procurement team running the standard bake-off would have eliminated it in round one.
This is not noise. The paper runs two-sided Wilcoxon signed-rank tests on paired valuation differences with Benjamini–Hochberg correction: Gemini's gain is significant at q = 1.28×10⁻¹⁵, DeepSeek's decline at q = .0116, Qwen's at q = .0061. GPT-5.5, Doubao and Claude move within noise. Three of six models change materially when the opponents change; three do not.
Why the Rankings Barely Correlate at All
Aggregate rank tables understate this badly, because the disagreement is at the task level and averaging hides it.
Across the 100 problems, the mean within-problem Spearman rank correlation between the two ecologies is 0.182, with a bootstrap confidence interval of [0.090, 0.273]. In plain terms: knowing how the six models ranked on a problem in Solo tells you almost nothing about how they rank on the same problem in Arena. The single largest movement is 35 problems won by DeepSeek in Solo that are won by Gemini in Arena.
The mechanism is intuitive once stated. Solo rewards optimisation against a stationary environment: the rule-based opponents follow their policy regardless of what you do, so the best strategy is to model the demand curve and exploit it. Arena rewards something else — behaving well when everyone else is also adapting, when an aggressive price cut invites a response and a production build-out lands into a market that five other planners have also decided to flood. Those are different competencies. There is no reason a model that is excellent at the first should be excellent at the second, and this benchmark is evidence that it usually isn't.
Most enterprise agent evaluation is Solo by construction. τ-bench, the reference benchmark for tool-using agents in retail and airline domains, places one agent against a simulated user and a fixed set of APIs — there is no competing agent anywhere in the environment. The same is true of the coding and retrieval benchmarks most teams actually use for selection. That was a fair simplification when agents answered tickets. It stops being fair the moment the agent's actions move a price that somebody else's agent is watching.
Your Counterparties Are Already Agents
The contested ecology is not a research curiosity — the payments and procurement stack is being rebuilt to assume it.
Google announced the Agent Payments Protocol (AP2) on 16 September 2025 with more than 60 partner organisations including Mastercard, American Express, PayPal, Adyen, Worldpay, Salesforce and ServiceNow. Its own worked example is explicitly agent-versus-agent: a shopper's agent signals intent to a merchant, "whose own agent can respond by creating a custom, time-sensitive bundle offer." The protocol specification formalises the exchange with signed Checkout and Payment Mandates, and defines an Autonomous Mode in which the user sets constraints up front and the agent transacts on its own inside them. We covered the competing standards in Google's UCP push against OpenAI's protocol and AWS AgentCore Payments.
On the procurement side it is already running at scale. Walmart's supplier negotiations through Pactum reported a 3% average gain with payment terms extended by an average of 35 days, and 68% of engaged suppliers closing an agreement — a vendor's own published figures, so read them as such, but the deployment is real and the supplier on the other side has every incentive to bring its own agent to the next round. When both sides automate, the negotiation stops being an agent against a static counterparty and becomes exactly the Arena condition.
The test is simple. Ask whether your agent's actions change the environment it will face next round. Ticket triage, document extraction and code review: no — Solo evaluation is adequate and cheaper. Pricing, bidding, spot freight, ad buying, marketplace listing, supplier negotiation, capacity allocation: yes — and a Solo-only bake-off is not evidence about those.
The Harness Did 4,557 Things the Models Didn't
The second useful thing in this paper has nothing to do with rankings: it counts how often the scaffolding, not the model, produced the executable decision.
ERPBench logs execution interventions, and across the 7,200 decision rounds it recorded 4,557 events. Production-capacity skips — an attempted production item that could not be scheduled under available machine capacity, lot-size or material constraints — account for 3,761 of them (82.5%). Unavailable investment choices are 355 (7.8%). Fallback decisions, defined as recovery from missing or malformed model output into a valid default action, are 275 (6.0%). Production-quantity clamping is 118, and outright tracebacks 48.
Read that fallback number again. On 275 occasions the harness wrote the decision because the model did not produce a usable one — and those rounds still scored. Every agent evaluation you run has an equivalent: the retry, the schema repair, the default that fires when the tool call comes back malformed. If you are not logging it, part of what you are grading is your own scaffolding. The paper notes that models with fewer logged interventions tend to reach higher Arena valuations, while being careful to say the six-point relationship "admits substantial variation and is not inferential."
This is the same failure mode as scoring an agent's patch without scoring its scope, which is what we found in the SWE benchmark file-parity work, and the same reason pooling judgments rather than re-running them makes an evaluation reproducible. Instrument the harness or you are measuring it.
The Contested Round Is Where Collusion Becomes Visible
There is a compliance argument for the second round that has nothing to do with picking a winner: some behaviours only exist when there is someone to coordinate with.
Fish, Gonczarowski and Shorrer put LLM-based pricing agents into oligopoly settings and found they "quickly and autonomously reach supracompetitive prices and profits," with seemingly innocuous variations in the prompt substantially changing how far prices drifted above competitive levels. Their results extend to auctions. A Solo evaluation cannot surface any of this, because tacit coordination requires a counterparty capable of responding — the fixed rule-based opponent will not.
The enforcement surface is not hypothetical. On 24 November 2025 the Justice Department announced a proposed settlement with RealPage that would stop it using competitors' nonpublic, competitively sensitive information to set rents at runtime and limit model training to nonpublic data aged at least 12 months. No fine, no admission — and the final judgment is still pending Tunney Act review as of publication, with the court having entered only the stipulation in March 2026. Its terms run for seven years from entry, terminable at four: a durable set of constraints on how a pricing model may be built and what it may see. If your agent will price against the market, the contested round is the cheapest place to find out what it does when the other side moves, and the only place your compliance team can watch it happen before customers can.
What This Paper Does Not Show
Take the strongest version of the objection: this is one simulation, and it is not proof that Gemini is better at competition than DeepSeek.
The authors say so themselves. ERPBench is a controlled ERP-inspired simulation with a fixed 100-problem slice, a six-round horizon and one six-model Arena composition; results may differ under other ecologies, opponent sets, longer horizons or information regimes. More sharply, they warn that "model-family labels do not guarantee immutable or route-identical vendor snapshots across ecologies," so the contrasts hold under the observed service configuration rather than as route-invariant causal effects. A hosted endpoint is not a frozen artefact — the same caution that applies whenever a prompt technique stops paying after a model version bump.
And the Arena is its own artificial ecology. Six frontier models competing simultaneously is not your market either. Your competitors may run one LLM, five rule-based systems, or a procurement manager with a spreadsheet.
None of that weakens the operative claim, which is narrow and survives all of it: rankings from one competitive ecology do not transfer to another, and you cannot know which one your deployment resembles unless you test both. The paper's own recommendation is to report "not only terminal scores, but also whether model-selection conclusions transfer across competitive market ecologies and how executable decisions are produced."
Build the Contested Round
The benchmark is MIT-licensed and public, simulator and problem set included, so the first version of this is a fork rather than a build.
This Week:
- Classify every agent in your pipeline as stationary-environment or responsive-environment. Write the list down and get the sponsoring business owner to agree. If pricing, bidding, procurement or allocation is on the responsive side, its existing evaluation is now provisional.
- Pull the intervention log out of your current harness. Count retries, schema repairs, clamps and defaults per candidate model. If you cannot produce that number, that is this week's engineering task — the rest of the playbook depends on it.
- Re-read your last bake-off deck and mark every conclusion that came from a fixed environment. That is most of them.
This Month:
- Stand up a second round on the same scenarios. Two opponent sets, not one: rule-based policies mimicking your actual counterparties' known behaviour, and the rival candidate models themselves. ERPBench's split shows those two give different answers.
- Score paired, not aggregate. Same seeded scenarios in both ecologies, per-problem differences, a signed-rank test with multiple-comparison correction. A mean rank table would have hidden a 0.182 correlation behind a tidy leaderboard.
- Add an unprompted-coordination check. Run the contested round with pricing latitude and have compliance review the trajectories for parallel pricing before anything is signed.
Before You Sign:
- Make ecology transfer a contractual artefact. Ask the vendor for results in a contested setting or run it yourself; "leads on our benchmark" is a Solo claim until proven otherwise. Related: what to demand across frontier model selection on cost, latency and governance.
- Put the contested round in the regression gate, not the selection memo. Model endpoints move; the eval gate you already run in CI is where this belongs on every version bump.
The Bottom Line
Every discipline that evaluates competitive systems learned this the hard way. Trading firms do not backtest a strategy without modelling its own market impact. Ad-auction teams do not ship a bidder validated only against replayed logs. Both learned that a policy which is optimal against a recording is frequently terrible against a respondent — and both learned it by losing money first.
Enterprise AI is arriving at the same lesson from the other direction, with a benchmark shelf built almost entirely on recordings. τ-bench, SWE-bench and their descendants gave the field a shared yardstick, and they were the right instrument for agents that answered a question and stopped. They were never designed to tell you what happens when the environment answers back.
The second round costs a fraction of the first, because the scenarios already exist. Run it before the counterparty does.
Continue Reading
- Claude Matched the Patch. Qwen Overshot. Score the Scope.
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- 12 Models Wrote a Fake Crate. None Refused a Real One.
- Your Supervisor Rewrote the Task. The 'Do Not' Fell Off.
- Claude vs GPT vs Gemini: Stop Comparing Per-Token Prices
- Google UCP Beats OpenAI Protocol: Microsoft, Amazon, Meta Adopt