Claude Sonnet 4.6 and Claude Opus 4.6 are just over a point apart on SWE-bench Verified. On a new study of 14,922 agent trajectories they are 13 points apart on something no vendor scorecard shows: how often the agent's patch touches exactly the files the human fix touched. That second number is the one your reviewers pay for. A pass rate says the tests went green. It says nothing about how much code arrived with them — and at equal scores, two agents can hand you very different diffs.
What 14,922 Trajectories Actually Measured
The study profiles what a coding task demands, not what a benchmark calls it. arXiv:2609.01271, posted 1 September 2026 by Radin Shayanfar, Keheliya Gallaba and Ahmed E. Hassan, introduces what the authors call the Spread–Novelty–Centrality profile: a three-axis characterisation of a repository-level coding task.
Each axis is mechanical, which is why you can compute it on your own repo:
- Spread is how widely a change distributes across the codebase — normalised entropy over the per-file share of changes, plus a "radius" measuring how far apart the touched files sit in the directory tree.
- Novelty is additions as a share of additions plus deletions, so 1.0 is a purely additive change and low values are rewrites.
- Centrality is architectural significance, built from fan-in, fan-out, churn (the probability a randomly picked recent commit touched at least one of the same files) and "mass", which combines callable size with cyclomatic complexity.
The authors ran five benchmarks totalling 2,487 instances: SWE-bench Verified (500), SWE-Gym Lite (230), FEA-Bench (1,401), FeatBench (156) and FeatureBench (200). Six agent configurations went through them — Claude Haiku 4.5, Sonnet 4.6 and Opus 4.6 via Claude Code, and Qwen 3.5 at 9B, 27B and 397B-A17B via Qwen Code — all on the Harbor evaluation harness. Six configurations times 2,487 instances is 14,922, which tells you something the paper is quiet about: this is one run per instance, not an average of many.
A Benchmark Label Predicts Almost Nothing
Two benchmarks carrying the same label can demand opposite work, because the label describes the pull request while the curation pipeline describes the task. The paper's first finding is that every pair of the five benchmarks is statistically separated on at least two of the three axes, and it traces each separation to a specific curation decision.
Three of those five are commonly filed under feature implementation, and they are not the same job:
- FEA-Bench selects pull requests that add new components across 83 GitHub repositories. New components have no incoming imports, no prior history and little accreted complexity — so the benchmark sits at systematically low Centrality by construction.
- FeatBench "admits only PRs that modify existing functions without adding or deleting any", pushing its gold patches toward in-place rewrites and compressing Novelty at the opposite end.
- FeatureBench carves out the lines implementing a feature from a working repository and asks the agent to add them back, so its Novelty concentrates near 1.0 with a tight interquartile range. It draws 200 tasks from 24 repositories.
Even the issue-resolution pair separates. SWE-bench Verified keeps only cases whose PR introduces unit tests that flip from failing to passing; SWE-Gym uses the same formulation over repositories deliberately disjoint from SWE-bench's 2,294-instance pool. Same label, different repositories, different demand distribution.
The practical consequence for a bake-off: when a vendor quotes a feature-implementation number, the useful follow-up is not "what did you score" but "on which curation pipeline". A score on FEA-Bench's low-Centrality new components does not predict behaviour on the high-fan-in service at the centre of your platform.
Claude Matches Scope. Qwen Exceeds It.
At comparable pass rates, the two model families succeed in measurably different shapes — and that shape is stable enough to be called a signature. The paper scores every run by whether the agent's edited file set matches the gold patch's file set exactly (parity), covers it and more (superset), or falls short (subset).
For Claude, scope discipline improves with scale. "The parity share among resolved runs climbs on all three patch verbosity metrics, on files from 0.17 at S to 0.41 at M and 0.54 at L" — that is Haiku 4.5, Sonnet 4.6 and Opus 4.6 in order. For Qwen it does not improve at all: resolved runs "place most of their mass in the >2× band (0.6 to 0.7 on files), with a much lower parity share (≈0.2 on files) across scales". A bigger Qwen is not a tidier Qwen. It is a bigger Qwen that still overshoots, which matches what a 16-day autonomous Qwen coding run looked like when someone actually read the commit trace.
Both families share one failure mode: under-editing. Unresolved runs "sit deeper in the sub-parity bands" regardless of family or scale, so a suspiciously small diff is the signal worth alerting on, not a suspiciously large one.
Now put that against the numbers the vendors publish. Anthropic's Claude Haiku 4.5 announcement reports 73.3% on SWE-bench Verified, "averaged over 50 trials, no test-time compute, 128K thinking budget". For Claude Sonnet 4.6, the footnote reads "Our score was averaged over 10 trials. With a prompt modification, we saw a score of 80.2%". For Claude Opus 4.6, the equivalent footnote reads "Our score was averaged over 25 trials. With a prompt modification, we saw a score of 81.42%".
So the published gap between Sonnet 4.6 and Opus 4.6 is 1.2 points. The file-scope parity gap between them, on the same six-configuration study, is 13. One of those two numbers is on every slide in your evaluation deck. The other decides how much of your senior engineers' week goes into reading the output.
It also puts the headline in perspective. Across all five benchmarks, the paper reports Opus 4.6 resolving 1,083 of the 2,487 instances — 43.5%, against the ~81% it scores on SWE-bench Verified alone. That is the same lesson as grading a proof of concept on completion rate rather than an average score: pick a different denominator and the vendor's best number stops travelling.
Your Ticket Wording Shapes the Diff Size
How a task is phrased appears to change how much code the agent writes, on top of whatever the change itself requires. This is the finding with the shortest path to your backlog — and the one to hold most loosely, because the evidence is a comparison between benchmarks, and those benchmarks differ in far more than how much they hint.
FeatBench gives away nothing. Its authors state that "task inputs consist solely of abstract natural language descriptions, devoid of any code or structural hints". Agents on FeatBench produce the most verbose patches relative to the gold solution across all three verbosity metrics. FeatBench is also hard in absolute terms: its own authors report a best success rate of 29.94% and a documented tendency toward "aggressive implementation".
FeatureBench gives away a great deal — each task ships clear interface definitions, import paths and descriptions of expected behaviour — and agent patches there are the most compact of the five. Part of that is an artefact worth knowing about: 33.8% of FeatureBench's gold-patch lines are comments or docstrings that the agent can simply omit and still resolve the task, which inflates the gold rather than shrinking the agent. FEA-Bench sits between them, supplying the signatures and docstrings of new components as explicit hints, and lands near parity.
The same artefact runs the other way at the verbose end, and the paper does not adjust for it. Every one of these numbers is a ratio against the gold patch, and FeatBench's curation deliberately admits only in-place edits — which is close to the smallest gold patch a benchmark can manufacture. An agent that extracts one helper function blows that ratio without behaving any differently. So hint density is a plausible reading of the gap, not the only one.
The authors' own reading is careful and worth quoting: "the wording of the problem statement, not only the required change, shapes what the agent produces."
Read that as an operational lever. A ticket that names the module, the interface and the expected signature is a diff-size control. A ticket that says "make search faster" hands the agent a blank cheque on scope, and it will spend it. If you have already moved to machine-checkable specs in your ticket format, this is the second reason to: the first was token cost, and this one is review cost.
What a Bigger Diff Costs Your Reviewers
Review capacity, not inference spend, is what an oversized diff actually consumes — and the empirical baseline for human review is far smaller than anything an unhinted agent produces.
Google's study of approximately 9 million changes from more than 25,000 authors and reviewers between January 2014 and July 2016 found that "over 35% of the changes under consideration modify only a single file and about 90% modify fewer than 10 files. Over 10% of changes modify only a single line of code, and the median number of lines modified is 24." The median review latency for that whole process is under 4 hours. Small changes are not an accident there; they are the thing that makes fast review possible.
The same paper measures what happens when changes get big: "the average number of comments per change grows with the number of lines changed, reaching a peak of 12.5 comments per change for changes of about 1250 lines." More comments per change means more round trips, and round trips are wall-clock time.
SmartBear's account of its Cisco Systems study reaches a compatible conclusion from the defect side, with two caveats: it is a vendor's write-up of its own product's data, and the study ran to 2006 — 2,500 reviews of 3.2 million lines by 50 developers in a single Cisco product group, two decades before any of this tooling existed. Within those limits, developers developers "should review no more than 200 to 400 lines of code (LOC) at a time", a review of 200–400 LOC over 60 to 90 minutes "should yield 70-90% defect discovery", and there is a "significant drop in defect density at rates faster than 500 LOC per hour".
Now do the arithmetic on an agent whose resolved runs sit in the >2× band. You are not saving review time; you are moving spend from a metered API into unmetered senior-engineer hours. That is the mechanism behind Zalando's finding that agentic PRs got bigger while a third were auto-approved, and behind teams hitting 65 agent PRs a week without anyone getting time back.
Where This Study Stops Short
The honest version of this finding is narrower than the headline, and you should hold it at that width. Five limits matter.
All five benchmarks are Python-only, and they cover only issue resolution and feature implementation. Nothing here tells you how a Java monorepo migration or a Terraform change behaves.
The 14,922 figure is exactly six configurations times 2,487 instances, which means one run per instance. Anthropic averaged its own Opus 4.6 number over 25 trials and its Haiku 4.5 number over 50, precisely because single runs are noisy. A 0.41-versus-0.54 parity gap from unreplicated runs is a signal, not a measurement you should quote to a board.
The runs used the Harbor harness with Claude Code and Qwen Code, not each vendor's own evaluation scaffold, and Anthropic's published figures include a prompt modification. Scaffold differences move scope behaviour, so a like-for-like comparison against the vendor's number is not available here.
Parity also assumes the human patch is the right scope, and it is only one solution. A separate study of 4,892 patches from ten top-ranked agents on SWE-bench Verified found that even patches that passed the tests and resolved the issue "made different file and function modifications compared to the gold patches from repository developers" — which those authors read as a limit of the benchmark's test coverage, not as agent sloppiness. Some extra scope is a worse diff. Some of it is a helper the original author should have extracted. The metric cannot tell the two apart, which is the point: neither can anyone until a human reads it.
And the benchmark underneath the most-quoted score has its own error bars. Epoch AI, which runs SWE-bench Verified independently, notes an estimated error rate of 5–10% in the dataset and excludes 16 of the 500 samples (3.2%) that do not run reliably in its infrastructure; the whole set is drawn from 12 Python repositories. The 500-instance set itself exists because human annotators screened the original SWE-bench for underspecified problem statements and overly specific unit tests.
None of that makes 0.54 a service-level objective. It makes it a reason to measure the same thing on your own repository — the same lesson as a silent model swap invalidating an eval nobody re-ran. An external benchmark is a hypothesis about your codebase, never a result on it.
Two Columns to Add Before Your Next Bake-Off
This Week:
- Pull 30 merged pull requests from the last two quarters where a human fixed something you would now hand to an agent. Record the exact file set of each. That is your gold scope, and it costs an afternoon to build.
- Open your bug and feature ticket templates and check whether either has a required field naming the target file, module or interface. If not, you have no diff-size control at all — the agent is inferring your architecture from prose every time.
- Ask every vendor still in the running two questions their datasheet does not answer: which harness produced the quoted score, and over how many trials was it averaged.
This Month:
- Replay those 30 tickets through each candidate agent and score three columns per run, not one: resolved yes/no, file set versus gold (parity, superset or subset), and lines changed versus gold. Parity share on resolved runs is the column that will separate vendors your pass rate cannot.
- Add the runtime check the paper recommends to harness builders — flag any patch that lands far below the expected scope. Under-editing marks failure for both model families, so a small diff on a hard ticket is the alert, not the win.
- Run half your tickets hinted and half unhinted, and measure the diff-size delta on your own code. If the gap is large, your ticket standard is worth more than your model choice this quarter.
Before Renewal:
- Price review hours alongside tokens. A cheaper model that reliably exceeds gold scope by 2× is a transfer from your API line to your payroll line, and only one of those shows up in the AI budget.
- Route by Centrality. The paper's advice to engineers is to "delegate single-file, low-Centrality changes to smaller agents" and "reserve high-Centrality changes for stronger models and closer oversight". Compute fan-in on your own repo and make the routing rule explicit rather than leaving it to whoever opened the ticket.
- Write scope parity into the success criteria of the contract, next to pass rate — and make sure the reviewer is not the same vendor as the author.
The Bottom Line
Every prior generation of developer tooling was sold on a single throughput number and paid for somewhere else. Static analysis was sold on findings and paid for in triage. Test generation was sold on coverage and paid for in flaky suites. Coding agents are being sold on pass rate, and the invoice is arriving in review capacity — which is exactly the resource your senior engineers were already short of before any of this started.
The useful thing about this paper is that the second number is computable today. File-scope parity needs a git log, a list of gold patches and an afternoon. It is a cheaper measurement than most of what sits in a vendor evaluation, and it is the one that survives contact with your monorepo.
A pass rate tells you the agent finished. Scope tells you who cleans up.
Continue Reading
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Agent Teams Hit 65 PRs a Week. Nobody Got Time Back.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
- Haiku Burned More Tokens Than Sonnet. Spec It in Code.
- Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
- Best AI Code Review Tools for a Monorepo: Buy Precision
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate