A five-point gap on a thirty-task model bake-off is not a result. It is a rounding error with a slide deck attached. If your team ran an internal eval last quarter, declared a winner, and used that winner to justify a routing change or a capacity commit, there is a good chance the experiment could not have told you what you concluded from it.
That is the useful finding inside the Era by Eon Benchmark, submitted to arXiv on 9 September 2026 by a team at the cloud data company Eon. The benchmark itself is a fictional-enterprise simulator with exact ground truth. The part worth your time is that its authors did something almost nobody does: they printed the error bars next to the leaderboard. The error bars are about 30 percentage points wide, and once you look at them, the ranking mostly disappears.
Nine Models, 33 Questions, and a 30-Point Error Bar
The headline result is that eight of the nine models tested are statistically indistinguishable from each other. Nine models answered the same 33 questions three times each, and accuracy ranged from 42.4% to 76.8% — but as the abstract states outright, "three of 36 pairwise differences remained supported after correction."
The full results table is where it lands. Claude Opus 4.8 topped the board at 76.8%, with a 95% interval of [61.6, 90.9]. GPT-5.6-Sol came second at 69.7%, interval [54.5, 83.8]. Claude Opus 5 and Claude Sonnet 5 tied at 67.7%, GPT-5.5 at 66.7%, Claude Sonnet 4.6 at 63.6%, GPT-5.6-Luna at 56.6%, Qwen3-Coder-Next at 55.6%, and GPT-5.4-Mini last at 42.4% with an interval of [27.3, 58.6].
A confidence interval is the range of true values consistent with the data you collected. The top eight intervals on that board all overlap one another — every one of them contains the range 61.6% to 71.7%. The top model's plausible floor (61.6%) sits below the sixth-place model's point estimate (63.6%). That is not a close race — it is an unmeasured one.
The three comparisons that survived all point at the same loser. The paper reports them as Claude Opus 4.8 exceeding GPT-5.4-Mini by 34.3 points, GPT-5.6-Sol exceeding it by 27.3 points, and Claude Sonnet 4.6 exceeding it by 21.2 points. Note what is missing: Claude Opus 5 and Claude Sonnet 5 both scored higher than Sonnet 4.6 and neither shows up. The set of significant results is not even monotone in the scores, which is roughly what you would expect when the signal is thin. The paper does not name the statistical test or the correction method it applied, which is the one place its statistical transparency stops short.
Why Three Repeats Did Not Save the Experiment
Running each question three times fixes sampling noise, not sample size — and sample size is the binding constraint. The paper's own limitations section says it plainly: "Three repeats reduce model-generation noise but do not create more questions." And: "The benchmark contains only 33 questions, so small score differences may result from the choice of questions."
This is the distinction most internal evals get wrong. There are two sources of variance in a model score. One is the model's own stochasticity — same prompt, different answer. Repeats kill that. The other is question sampling: the 33 tasks you happened to write are a draw from the infinite set of tasks you could have written, and a different draw would produce a different ranking. Repeats do nothing to that, and it is usually the larger term.
Evan Miller's Adding Error Bars to Evals, published by Anthropic in November 2024, is still the best treatment of this and still largely ignored in practice. Its core move is to treat eval questions as drawn from an unseen super-population and apply the Central Limit Theorem, which is exactly the framing that makes a 33-question benchmark's interval come out at 30 points. That last step is contested, and not in the direction that would let you off the hook: a 2025 position paper, Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints, argues that below roughly a few hundred questions the normal approximation performs "very poorly," usually "dramatically underestimating uncertainty," and recommends Bayesian or exact frequentist intervals instead. Era never says how it computed its own. If they are the normal-approximation kind, 30 points is the generous reading. Miller's paper also shows what happens when questions cluster — when they share a template, a document, or a source system: on the DROP benchmark, "clustered standard errors can be over 3X larger than naive standard errors." Most enterprise eval sets are heavily clustered by construction, because they are built from a handful of real tickets or a handful of real documents.
If your eval set is 30 tasks generated from five source documents, your effective sample size is closer to five than thirty. That is not a technicality. That is the whole measurement.
The Averages Hide the Only Number That Matters
The aggregate score is the least informative number in the table, and the task-type breakdown is the most. On the Era benchmark, models hit 92.6% on simple filtering questions, 39.6% on cross-system questions, and 3.7% on multi-hop questions.
Three point seven percent. Not a degradation — a floor. The 33-question set splits into 21 single-system questions and 12 composed questions that span products through join keys — "the recorded call minutes of the account with the largest open pipeline" is one of the paper's own examples — with two of those twelve unanswerable, checking whether an agent will admit that the information is absent. An enterprise agent that answers simple filtering questions at 93% and multi-hop ones at 3.7% is not a 63%-accurate agent. It is two different products, and which one you bought depends entirely on which questions your users actually ask.
This is the same trap as scoring a proof of concept on an average. We covered it when StartupBench agents averaged 73 while only 30% of their output was usable, and it recurs every time a vendor leads with a single blended number. If your eval reports one figure, it is hiding the shape of the failure, and the shape of the failure is the thing that determines whether the agent ships.
Worth noting how hard the environment is. The harness exposes roughly 290 MCP tools across simulators of 66 products — Salesforce, Zendesk, Slack, Jira, Gong, Stripe, S3, HubSpot, Google Drive — and, per the paper, "nothing about this surface is described in the model's prompt: no schemas, no data description, no tool list." Agents get 25 turns, 40 tool calls, and 8,000 characters per tool response. Discovery is part of the task. That is a fair approximation of what an agent faces against a real estate, and it is why the multi-hop number is so low.
How Many Tasks You Actually Need — and What They Cost
Intervals narrow with the square root of the question count, so cutting a 30-point interval in half takes four times the questions and cutting it to a quarter takes sixteen times. Concretely: 33 questions buys you ±15 points. Roughly 130 questions buys you ±7. Roughly 530 questions buys you ±4 — which is the first point on that curve where a five-point leaderboard gap starts to mean something. Those counts assume the tasks are independent draws. Clustered ones are worth less than their headcount, so treat the numbers as a floor and not a target.
That is not an unreasonable ask, and the industry already builds at that scale. SWE-bench Verified is 500 human-filtered instances. GAIA is 466 questions with 300 held back for the leaderboard. Those numbers are not arbitrary; they are roughly where a benchmark stops being a demo.
The cost argument against doing this has quietly collapsed. The Era comparison ran 891 total runs — nine models, 33 questions, three repeats — for $343.27 in inference. That is about 39 cents a run — but that is a blend across all nine models including the two cheapest on the board, and the paper publishes no per-model breakdown, so budget above it if your shortlist is two frontier models. Two models, 500 questions, three repeats each is 3,000 runs: somewhere between $1,200 at Era's blended rate and $5,500 at the $1.84 a rollout the Holistic Agent Leaderboard paid to run 21,730 rollouts across nine models and nine benchmarks for about $40,000. Either end of that range is a rounding error against the commit you are about to sign.
The expensive part was never tokens. It is writing 500 grounded tasks with verifiable answers, and that is a data-authoring problem your domain experts have to sit through. Which is precisely why the pooled-judgment approach matters — JPMorganChase ranked 62 retrieval configurations for about $800 by reusing judgments across runs instead of re-grading from scratch each time. Build the answer key once, amortise it across every model you will ever test.
The Case for Ignoring the Error Bars
The strongest counterargument is that wide intervals are an artifact of this benchmark's size, not evidence that model differences are imaginary — and it is a fair one. A 34-point gap did survive. Nobody sane concludes from this paper that GPT-5.4-Mini is as good as Claude Opus 4.8. The finding is about resolution, not about whether models differ.
The second counterargument is that statistical significance is the wrong bar for a business decision anyway. You are not publishing a paper; you are choosing between two vendors, and choosing the one that scored higher on a noisy signal is still better than choosing at random. True, as far as it goes. But it stops going the moment the decision has a switching cost — a migration, a re-write of your prompt library, a twelve-month commit. A coin flip you have to live with for a year is not a decision, and the honest framing is that the eval abstained.
Third, and most cynical: leaderboards have deeper problems than sample size. The Leaderboard Illusion documented 27 private LLM variants tested by Meta before the Llama-4 release, and Arena data allocations of roughly 19.2% to Google and 20.4% to OpenAI against roughly 29.7% shared across 83 open-weight models — asymmetries the authors link to relative performance gains of up to 112%. Separately, researchers showed that dropping just 0.003% of Chatbot Arena preferences can change which model ranks first. And a 2026 paper on agent evaluation argues that rankings from aggregate scores do not transfer out of distribution at all. Sample size is the cheapest of these problems to fix. It is the only one you control.
One structural caveat about Era itself: the generator is not released. The paper describes it as "an internal development tool: clients receive the finished estate and answer keys, not the generator itself." A benchmark you cannot regenerate is a benchmark you cannot extend. And the interest is worth stating plainly: Eon sells Eon Mind, a layer that maintains unified live context across a customer's systems and exposes it to agents over MCP. A result showing that agents collapse to 3.7% when forced to discover and join across 66 systems unaided is a result that flatters exactly what Eon sells. That does not make it wrong — the answer keys are computed, and every number here is checkable against the paper — but it is why a vendor-published benchmark stays a vendor's claim until someone independent reproduces it, the same standard we applied when DeepSeek quietly swapped a model's post-training and every downstream eval silently went stale.
What to Do Before Your Next Model Decision
This Week: Pull the last model bake-off your team ran and write down two numbers — how many distinct tasks it contained, and how many independent source documents or systems those tasks were drawn from. If the first number is under 100 or the second is under 20, the result did not separate the models and you should stop citing it in architecture reviews. Then recompute your headline gap as a paired difference between the two models on the same tasks rather than two independent averages; in Miller's worked example, pairing alone cuts estimator variance by a third.
This Month: Get to 200 tasks minimum, with the answer key computed rather than hand-written, and tag every task by shape — single-system lookup, cross-system join, multi-hop reasoning, unanswerable. Report per-shape accuracy, never a blended average. Budget the authoring time, not the token spend; at roughly 39 cents a run the compute is a rounding error against a single engineer-week. Add repeats only after you have the task count, because three samples of the same question tell you about the model's variance, not the world's.
Before Renewal: Adopt a decision rule in writing and get it signed by whoever owns the budget. Ours: if the intervals overlap, the eval abstains and the choice falls to cost per resolved task, p95 latency, data-retention terms, and exit rights. Write into the contract that a material change to the served model triggers a re-run of your suite — because the vendor's version number is not a version number, and a silent post-training update invalidates every number you collected. Score the exit, not the leaderboard, exactly as you would when selecting an agent orchestration platform.
Ongoing: Wire the suite into CI so the regression gate runs on every model or prompt change. The tooling is mature and mostly free at this scale — Braintrust, Langfuse and Promptfoo all handle dataset versioning and paired diffs, and none of them will stop you from drawing conclusions from 30 rows, so the discipline has to come from you.
The Bottom Line
Enterprise IT has been here before. Vendors shipped whatever database or systems benchmark number flattered them, and buyers eventually learned that a benchmark without a published methodology and a stated margin of error was marketing. The industry built standards councils in response — SPEC and the Transaction Processing Performance Council were both founded in 1988, the latter to "disseminate objective, verifiable" performance data. AI evaluation is at the pre-council stage: point estimates on small samples, no error bars, no correction for multiple comparisons, and a leaderboard refreshed weekly.
The Era paper is worth reading not because the benchmark is definitive — it is 33 questions from a vendor with an unreleased generator — but because it did the one thing the rest of the field skips. It showed its uncertainty. Nine models, 33 questions, three repeats, $343.27, and an honest admission that 33 of 36 comparisons could not be called. That is what a credible eval looks like when it does not have enough data, and it is a better artifact than a confident ranking built on the same 33 questions would have been.
Your bake-off has error bars too. You just have not calculated them.
Continue Reading
- DeepSeek Won Solo. Gemini Won the Arena. Run Both Rounds.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- Eleven Voice Agents, One Bank Call, No Clean Winner
- Agent Orchestration Platforms: Score Exit, Not Features