Your merge gate does not test what your reviewers test. It runs the suite, sees green, and lets the patch through — and roughly a third of the agent patches it approves are ones your own reviewers would have sent back.
That is a measured number now, not a suspicion. SWE-Gate, a repository-level benchmark submitted to arXiv on 3 September 2026 by Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang and Guanbin Li, scores review-constraint compliance separately from functional correctness. Across four models the result was blunt: "among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints." That is a 34.3% hidden failure rate on SWE-Gate's own tasks — and the blind spot it exposes sits inside every functional-only agent scorecard you have ever been shown, including the ones in vendor launch posts, because none of them measure the constraint at all.
If you report "agent success rate" to a steering committee as the CI pass rate, you are overstating mergeable output by roughly 22 points.
What a Review Constraint Actually Is
A review constraint is an acceptance requirement a human reviewer states in a pull request comment that the issue's own tests never check — raise a KeyError instead of returning None, do not change the public schema, keep the argument order, do not widen the fix beyond the reported case.
SWE-Gate is built entirely out of those. The authors mined merged pull requests through the GitHub API, used an LLM to extract "atomic suggestions explicitly stated in review comments," then applied deterministic filtering and semantic validation to keep only constraints that could be turned into an executable test. The finished benchmark is 303 repository-level repair instances across 75 open-source Python repositories.
Each instance ships four artifacts, and the relationship between them is the whole design. A mutant.patch injects "a reproducible target defect into the original repository while preserving surrounding engineering context." A functional test checks the bug is fixed. A constraint test is "an executable test that validates compliance with the review constraint." And a non-compliant patch is "a reference repair that passes the functional test but violates the constraint" — proof that the two tests are genuinely independent. A Docker-based validation matrix enforces it: the mutant must fail the functional test, the non-compliant patch must pass functional and fail constraint, and the gold patch must pass both.
The paper's worked example is a Pydantic pull request where the reviewer asked for a new extras_ser_exclude_if parameter rather than a change to the public schema. The functional test checks that qualifying extra fields are omitted from serialized output. The compatibility test checks that the existing public schema is unchanged. An agent can satisfy the first by breaking the second, and nothing in a conventional harness would notice.
All of it is released on GitHub — the 303 instances, the per-instance test suites, gold and non-compliant patches, 99 reproducible Docker base images, and an evaluation harness that runs without any LLM calls. You can point it at your own model choice this week.
The 34.3% Your Scorecard Never Shows
The gap between functional and joint success is not a rounding error at any capability level. All four backends ran under the same scaffold, with the constraint supplied in the prompt:
| Model | Functional passes (of 303) | Joint passes | Functional rate | Constraint-following | Joint rate |
|---|---|---|---|---|---|
| GPT-5.5 | 227 | 160 | 74.9% | 70.5% | 52.8% |
| DeepSeek-V4-Flash | 202 | 130 | 66.7% | 64.4% | 42.9% |
| GPT-5.4-mini | 187 | 120 | 61.7% | 64.2% | 39.6% |
| GPT-4o-mini | 28 | 13 | 9.2% | 46.4% | 4.3% |
The hidden-failure column is the one to take to your next planning meeting: 67 of GPT-5.5's 227 functional passes (29.5%) failed a constraint, along with 72 of 202 for DeepSeek-V4-Flash (35.6%), 67 of 187 for GPT-5.4-mini (35.8%) and 15 of 28 for GPT-4o-mini (53.6%). Getting a better model narrowed the gap. It did not close it.
Now notice what kind of number you have already seen. Every run used Mini-SWE-Agent, a scaffold of roughly 100 lines of Python whose README states it "scores >74% on the SWE-bench verified benchmark" — the class of figure that fills leaderboards and launch posts. GPT-5.5's functional rate here was 74.9%. Different benchmark, and the README attaches no model to its number, so treat the resemblance as coincidence rather than a matched pair. What matters is what a figure of that kind counts. It was never wrong. It was the numerator of a fraction whose denominator changed the moment somebody asked whether the patch was acceptable rather than merely correct.
That distinction is baked into the older benchmark's own description. SWE-bench Verified is "a subset of 500 problems that real software engineers have confirmed are solvable" — confirmed solvable, which is not the same as confirmed mergeable. Vendors then compress even that into a claim about the job: Anthropic's November 2025 Claude Opus 4.5 announcement described the model as "state-of-the-art on tests of real-world software engineering" above a SWE-bench Verified chart. Real-world software engineering includes the reviewer. The test does not.
This is the same measurement problem from a different side than the file-parity work that scored how wide an agent's diff runs versus the human patch. That one measures the review workload an agent creates. SWE-Gate measures whether the work it already finished is acceptable.
Writing the Rules Down Beat the Model Upgrade
The most useful table in the paper is the ablation, because it is the only one that tells you where to spend money. The authors ran every model twice: once with the constraint withheld (−C) and once with it supplied as a natural-language sentence alongside the issue (+C).
| Model | Joint rate, constraint withheld | Joint rate, constraint supplied | Constraint-following change |
|---|---|---|---|
| GPT-5.5 | 41.3% | 52.8% | +15.9 pts |
| DeepSeek-V4-Flash | 38.0% | 42.9% | +10.2 pts |
| GPT-5.4-mini | 36.3% | 39.6% | +13.5 pts |
| GPT-4o-mini | 3.3% | 4.3% | +25.6 pts |
Providing the constraint raised joint success for every model, by 11.5 points on the strongest backend. But run your eye across the two middle columns instead of down them. DeepSeek-V4-Flash with the rules in context scored 42.9% — higher than GPT-5.5 without them at 41.3%. Same scaffold, same 303 tasks, same week. On this data, writing your review constraints down and shipping them in the prompt bought more mergeable output than moving up a model tier.
The honest cost is in the column I left out. Functional success fell for all four models under constraint guidance — 75.6% to 74.9% for GPT-5.5, 70.0% to 66.7% for DeepSeek-V4-Flash, 71.6% to 61.7% for GPT-5.4-mini, and 15.8% to 9.2% for GPT-4o-mini. Attention spent honouring a constraint is attention not spent on the bug, and the smaller the model, the more that trade hurts. The joint rate still improved in every case, which is the only score that corresponds to a merged pull request. But if your internal dashboard tracks functional pass rate, adding a constraints file will look like a regression on the way to being an improvement. Change the dashboard first.
This is the empirical version of an argument that has been circling agent tooling for a year: specifying work in machine-checkable form changes agent behaviour more reliably than prose does. SWE-Gate puts a number on the prose half of it.
The Constraints Agents Miss Most
Two categories dominate, and neither is exotic. Across the 303 instances, error semantics appears in 152 (50.2%) and schema, metadata or typing in 143 (47.2%), followed by ordering and argument preservation (86), encoding, escaping and quoting (74), scope generalization (62), compatibility and deprecation (55), sentinel distinction (51), performance and structure (41), idempotence (30), and lifecycle or resource cleanup (19). Instances carry more than one constraint, which is why the shares exceed 100%.
Compliance is worst on the ones that require the agent to not do something. Across the three capable backends, scope generalization ran 46–63% compliance, lifecycle cleanup 54–63%, and encoding or escaping 51–68%. A model asked to fix a bug in one code path will happily generalise the fix across three more, close a file handle in the wrong place, or normalise a string on the way through — each of which is a perfectly reasonable engineering instinct and a review comment waiting to happen.
Look at that list again and ask which of them your existing tooling catches. A type checker catches some typing drift. A linter catches nothing about whether an exception type is the one the API contract promises, whether a fix stayed inside its blast radius, or whether a public schema is still the public schema. These are project-specific acceptance rules that live in reviewers' heads and in five years of pull request threads — which is exactly where SWE-Gate went to find them, and exactly where you should go to find yours.
It also sharpens a governance problem this publication has flagged before. If the same vendor's model is both writing and reviewing the pull request, and both sides share the blind spots measured here, the second pass is not independent evidence of anything. Nor is a fast lane: teams auto-approving a large share of agent PRs are auto-approving this 34.3% along with everything else.
Where This Benchmark Stops Short
Take the result seriously and the framing carefully. Five limits matter before you cite this in a budget conversation.
It is Python only, and small. 303 instances across 75 repositories. The authors say plainly that "future work should extend SWE-Gate beyond Python and develop reliable evaluation methods for review requirements that cannot yet be expressed as executable tests." The constraints that survived into this benchmark are the ones a test could check. The ones your reviewers argue about hardest are often the ones a test cannot.
The instances are synthesized, not observed. Each task starts from a mutant patch that injects a defect into a real repository. That buys reproducibility and a clean validation matrix; it does not prove the same rates hold on organically reported issues.
Not every released instance is fully reconstructed. The repository documents 255 instances with complete validation matrices and 48 that lack reconstructed optional files. If you rerun this yourself, check dataset_info.json before quoting a headline rate.
One scaffold, one shot. Everything ran on Mini-SWE-Agent with at most 100 interaction steps. A harness with a self-review pass, a linter in the loop, or a second model checking the diff could close part of this gap — and that possibility is untested here, which makes it an argument for building one rather than for dismissing the finding.
And the constraint-provided condition flatters you. The agent receives a clean, single-sentence, pre-validated natural-language constraint. Your review history is not clean sentences; it is 4,000 comments of varying temper, half of them obsolete. The +C column is closer to a best case than to your repository on Monday morning. Which is the point — the gap is at least this wide.
One column deserves outright suspicion: GPT-4o-mini's 46.4% constraint-following rate is computed over 28 patches. At a 9.2% functional rate that model barely reached the starting line, and its +25.6-point constraint gain is measured on a handful of cases. Do not build a cheap-model strategy on that row. Build it on DeepSeek-V4-Flash's 202.
What to Do Before Your Next Model Upgrade
This Week:
- Pull 90 days of merged PR review comments and classify 100 of them by hand. Split into "the tests would have caught this" and "no test in this repo checks this." The second bucket's share is your local version of 34.3%, and it takes an afternoon. You cannot argue about a benchmark's external validity once you have your own number.
- Stop reporting agent success as the CI pass rate. Report two figures instead: merged with no human-requested change, and merged after at least one. Every tool in the chain — GitHub Copilot, Cursor, an AI review bot — already writes both events into your PR history.
This Month:
- Write your top ten recurring review comments into a constraints file the agent actually reads. AGENTS.md is the open format, "a README for agents," and is used by over 60,000 open-source projects; GitHub's own repository custom instructions live at
.github/copilot-instructions.mdand are honoured by the Copilot cloud agent and Copilot code review. Note GitHub's guidance that instructions "must be no longer than 2 pages" and "must not be task specific" — the file is a ranked shortlist, not an archive, so order it by how often a human repeats a comment, not by how much the rule matters in theory. - Promote the three most-repeated constraints from prose to executable checks. SWE-Gate's construction is the template: if a rule can be a test, it should be a test, and then a required status check on the protected branch rather than a sentence someone hopes the model reads. Rulesets let you layer several of these on one branch and flip enforcement without deleting the rule, which is how you pilot a new constraint without blocking the org.
- Re-tier what gets human eyes. Constraint-heavy changes — public interfaces, error contracts, serialization, anything touching a schema — go to a human. The rest can ride the gate. This is the same review-tiering logic that applies to agent instruction files themselves, and it is cheaper than adding reviewers.
Before Your Next Upgrade Decision:
- Run the cheaper model with your constraints file against the expensive model without one. On SWE-Gate that configuration won. If it wins on your repository too, you have just funded the constraints work out of the model budget — and if it loses, you have a real number to justify the upgrade instead of a vendor chart. Your existing eval harness can carry the comparison; you need one new column, not a new platform.
The Bottom Line
We spent fifteen years making the merge gate mechanical, and somewhere in that project the working definition of "correct" quietly shrank to "whatever a test asserts." That was survivable when a human wrote the patch, because the same human held the unwritten rules and a reviewer held them too. Agents did not create the gap between passing and acceptable. They industrialised it, at machine cadence, on a gate that was only ever built to check the half we bothered to encode.
The fix is unglamorous and it is not a purchase order. It is an afternoon reading your own review comments, a two-page file, and three new tests.
Green is not the same as mergeable. Write down the difference, or keep paying a reviewer to remember it.
Continue Reading
Claude Matched the Patch. Qwen Overshot. Score the Scope.
Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
Haiku Burned More Tokens Than Sonnet. Spec It in Code.
12 Models Wrote a Fake Crate. None Refused a Real One.
Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.