Choose a coding agent by what one working change costs, not by who tops the leaderboard. On Real-SWE, a new benchmark built on licensed private company codebases, Fable 5.1 resolved the most tasks at 38.8% but works out to about $17.94 per resolved task, while Gemini 3.8 Flash resolved 31.2% for about $8.01. The models also failed in different ways — so a single generic review checklist misses most of what your chosen agent actually gets wrong.
The sample behind those numbers is small, and its codebases cannot be inspected. That makes the gap between the top three models noise you should not buy on. The cost arithmetic and the per-model failure breakdown are a different matter: both are usable now, as long as you rerun them on your own code before a renewal.
What Real-SWE Actually Measured
Real-SWE measures model-and-harness pairs on long, multi-file tasks taken from production codebases whose owners licensed them to Specific Labs, a startup whose site says it is backed by Y Combinator. The benchmark page, credited to Snagnik Das, Siddhant Paliwal and Janak Sunil, describes the source companies as "real companies with substantial usage, strong engineering teams, and demanding production workloads" — among them a Luma/Partiful competitor with 200K+ users, a consumer fintech platform processing 100K+ bank statements, and enterprise AI sales platforms.
The tasks look like your backlog, not a public issue tracker. Per the page, the median instruction runs 1,742 characters and the median reference solution edits 11 files, against 6 for FrontierCode and DeepSWE. The stack spans Go, Python, Node.js and TypeScript, with AWS, Docker, Kubernetes, PostgreSQL, MySQL, MongoDB and Redis. Verifiers are "inspired by existing test suites in the codebase or use those tests verbatim," and scores are "pass@1, averaged over eight independent runs per task." Spanish-language startup press covered the release on September 12.
Each model ran in its maker's own agent harness: Fable 5.1 in Claude Code, GPT-6 Astra and GPT-5.6 Sol in Codex CLI, Gemini 3.8 Flash in Gemini CLI, Grok 4.6 in Grok Build, Muse Spark 1.3 in Muse Code and Kimi K3 in Kimi Code. The exception is Zai's GLM 5.3, which ran in Claude Code. Replying in a Hacker News thread that drew more than 230 points and 130 comments, Sunil wrote that "all models were run on 'high' reasoning" and "we've done our best to use the native provider's harness."
For a buyer, that harness choice is a feature rather than a flaw. You never deploy a bare model; you deploy a model inside a tool. What the result cannot tell you is how Fable 5.1 would do inside Cursor, or Gemini 3.8 Flash inside Claude Code.
Why private code, and why this isn't the first try
Public coding benchmarks have a memorisation problem. OpenAI audited 138 SWE-bench Verified tasks that GPT-5.2 consistently failed, concluded that 59.4% of them were broken, and found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce exact fixes from memory, Decrypt reported in February. Specific Labs' pitch is that "tasks on private codebases are natively out of distribution."
Real-SWE is not the first benchmark to act on that. Scale AI's SWE-Bench Pro, published in September 2025, included a commercial subset of 276 instances from 18 startup codebases. Claude Opus 4.1 fell from 22.7% on the public set to 17.8% on the commercial one, and GPT-5 from 23.1% to 14.9%. What Real-SWE adds is current models, longer tasks and — the part worth copying — cost and failure data for every model.
Read it knowing who wrote it. Specific Labs' business is turning real company data into datasets for building agents, so a benchmark showing frontier models struggling on private code doubles as a sales argument. That does not make the numbers wrong. It is a reason to rerun them on your own code.
Cost per Resolved Task Reorders the Leaderboard
Divide each model's estimated cost per rollout by its resolution rate and the leaderboard nearly inverts: Gemini 3.8 Flash becomes the cheapest source of a working change, and Fable 5.1 one of the most expensive. Cost per resolved task is the spend per attempt divided by the share of attempts that pass — what you pay, on average, for one change that works.
| Model | Harness | Resolution rate | Est. cost per rollout | Cost per resolved task |
|---|---|---|---|---|
| Gemini 3.8 Flash | Gemini CLI | 31.2% | $2.50 | $8.01 |
| Muse Spark 1.3 | Muse Code | 23.8% | $2.74 | $11.51 |
| GPT-6 Astra | Codex CLI | 33.8% | $4.67 | $13.82 |
| Grok 4.6 | Grok Build | 23.8% | $3.44 | $14.45 |
| GPT-5.6 Sol | Codex CLI | 16.2% | $2.65 | $16.36 |
| GLM 5.3 | Claude Code | 28.8% | $5.12 | $17.78 |
| Fable 5.1 | Claude Code | 38.8% | $6.96 | $17.94 |
| Kimi K3 | Kimi Code | 18.8% | $3.90 | $20.74 |
The first four columns come from the Real-SWE leaderboard; the last divides one by the other. The page itself notes that "higher cost does not guarantee a higher resolution rate," but it does not say how cost per rollout was estimated — including whether prompt caching was counted. On agentic coding work, cache reads can dominate the bill, so treat the dollar column as directional. Time does not buy correctness either: per the page, 71.4% of rollouts under ten minutes failed, against 73.4% of longer ones.
The strongest case for the expensive model
The metric ignores your people. A failed attempt is not free — someone has to notice that it failed. To land ten resolved tasks, Fable 5.1 needs about 25.8 attempts and Gemini 3.8 Flash about 32.1. Flash saves roughly $99 in model spend and hands your reviewers about six more failed attempts.
If catching one failed attempt costs your team more than about $16 of engineer time, Fable 5.1 is the cheaper agent; if it costs less, Flash is. Against GPT-6 Astra, Fable's break-even drops to about $11. For an 11-file change to a production service, that is a low bar.
So the real decision variable is how cheaply your pipeline rejects a bad change before a human reads it. A strong CI suite that fails wrong changes automatically pushes the break-even up and favours the cheaper model. A thin one favours the more accurate model. It is the same trade-off that decides whether a model router pays for itself: the task mix, not the rate card, sets the answer.
Each Model Fails in a Different Way
Real-SWE's failure breakdown shows each model has a dominant way of getting it wrong, so the review check that catches the most failures depends on which agent you run. The page groups failed runs "using the same taxonomy across models, following DeepSWE," with five categories:
- Unverified assumption: "Builds on a guess about the system instead of checking it in the workspace."
- Missed requirement: "Leaves out behavior the instruction requires."
- Integration error: "Right idea, wired into the surrounding system incorrectly."
- Regression: "Breaks existing behavior while making the change."
- Wrong file: "Delivers the change somewhere the running application never calls."
Share of each model's failed runs, per the benchmark page:
| Model | Failed runs | Missed requirement | Integration error | Unverified assumption | Regression | Wrong file |
|---|---|---|---|---|---|---|
| Fable 5.1 | 49 | 36.7% | 34.7% | 24.5% | 4.1% | 0% |
| GPT-6 Astra | 53 | 28.3% | 34.0% | 34.0% | 3.8% | 0% |
| Gemini 3.8 Flash | 55 | 29.1% | 49.1% | 10.9% | 10.9% | 0% |
| GLM 5.3 | 57 | 38.6% | 26.3% | 28.1% | 0% | 7.0% |
| Grok 4.6 | 61 | 67.2% | 8.2% | 24.6% | 0% | 0% |
| Muse Spark 1.3 | 61 | 36.1% | 41.0% | 19.7% | 3.3% | 0% |
| Kimi K3 | 65 | 53.8% | 27.7% | 15.4% | 0% | 3.1% |
| GPT-5.6 Sol | 67 | 31.3% | 16.4% | 43.3% | 9.0% | 0% |
Three profiles stand out:
- Grok 4.6 builds the wrong subset of the right thing. 41 of its 61 failed runs left out required behaviour, and only 8.2% were integration errors. Kimi K3 leans the same way at 53.8%.
- Gemini 3.8 Flash has the right idea and wires it in wrong. 27 of its 55 failures were integration errors, and it has the highest regression share in the table at 10.9%. That is the class of mistake unit tests pass and integration tests catch.
- GPT-5.6 Sol guesses instead of looking. 29 of its 67 failures were unverified assumptions about the system.
Fable 5.1 and GPT-6 Astra have no single mode above 37%. Their errors spread across the three big categories, which means no one check covers them — the price of the best resolution rates is the least predictable failure.
Three cautions. The page does not say whether people or a model assigned the labels. Each share comes from 49 to 67 failed runs on the same small task set, so one task that trips a model eight times can set its profile. And one Hacker News commenter read the example instructions as "poorly specified," producing failures "due to bad assumptions and missing requirements" — if so, those two categories partly measure the tickets rather than the models, and the usable signal is the difference between models on the same tasks. That is why the next step is to measure your own, which is also where Tess Ainsley lands in a write-up: "measure the acceptance rate per agent, per area of the codebase, and adjust the review depth from that, per path."
Key Your Review Gate to the Failure Mode
Turn each dominant failure category into one specific, checkable gate on agent pull requests, instead of asking reviewers to look for everything at once.
- Missed requirement → requirement traceability. For Grok 4.6 and Kimi K3, have the agent restate the acceptance criteria as a checklist in the PR description and point to the diff hunk that satisfies each item. Reviewers check the list before the code. Put the rules the change must honour into the agent's context, not only the ticket — the lesson of 221 green patches that failed review.
- Integration error → tests at the call site. For Gemini 3.8 Flash and Muse Spark 1.3, require the PR to run the service-level or end-to-end suite, not just the unit tests near the edit, and flag any changed interface whose callers the diff did not touch. Flash's regression share argues for the full regression suite on every agent PR.
- Unverified assumption → evidence of inspection. For GPT-5.6 Sol, and the third of GPT-6 Astra's failures in this category, require the PR to cite what the agent checked behind each claim about the system: the schema it read, the config value, the command output. An assumption with no trace goes back.
- Wrong file → reachability. Rare — GLM 5.3 at 7.0%, Kimi K3 at 3.1% — but cheap to catch. Require coverage on the changed lines; code the application never calls cannot get any. Watching diff scope helps too, since models differ in how far their patches sprawl.
If you run more than one agent, the gate follows the agent, not the repository. That is also an argument against letting one vendor's model review its own agent's output: the reviewer is likely to share the blind spot you are trying to catch.
How Much Should You Trust This Leaderboard?
Trust the cost arithmetic and the failure breakdown as hypotheses to test on your own code; do not trust the order of the top three. The page never states how many tasks the leaderboard covers; it introduces its analysis as "a small sample of tasks from our benchmark," and that breakdown shows ten tasks, run eight times each. The leaderboard reconciles exactly to those ten — the page's duration statistics count 640 rollouts, eight models times 80 — and so do the failed-run counts: Fable 5.1's 49 failures against a 38.8% rate imply 80 rollouts and 31 resolved, Gemini 3.8 Flash resolved 25, and GPT-6 Astra 27. First and third place are six resolved rollouts apart.
Even treating all 80 rollouts as independent, a 38.8% rate carries a 95% interval of roughly ±11 points — wide enough to contain Flash's 31.2%. They are not independent. Eight attempts at one task share that task's difficulty, and Evan Miller's Adding Error Bars to Evals warns that "clustered standard errors can be over 3X larger than naive standard errors." More reruns cannot fix it: resampling cuts noise within a question but not "the variance of the conditional mean." The effective sample here is closer to ten tasks than to 80 runs. The page says "95% confidence intervals are shown" on its chart, but does not state the bounds as numbers in its text.
Miller's remedy is also what to ask for: score two models task by task on the same questions, because "paired differences represent a 'free' reduction in estimator variance." It is the same discipline that showed only 3 of 36 model gaps were real on another small eval set.
Hacker News commenters found the same weaknesses. "So TL;DR benchmarking in a completely non-reproducible manner?" asked one; another said the write-up was missing "crucial details... like reasoning levels," others noted it does not measure speed — which matters for an agent sitting in a developer's inner loop — and some said Gemini 3.8 Flash's third place did not match their own experience with it. Sunil replied that "we manually vet all codebases and companies" and that "we're going to open source some of our tasks and model trajectories." Until that happens, nobody outside Specific Labs can check a single score, and a score on one model version can go stale the day a vendor swaps the weights.
What to Do Before Your Next Agent Renewal
Rebuild the Real-SWE measurement on your own code, at a sample size that can separate the models, before you sign anything.
This Week:
- Compute your current agent's cost per resolved task. Take last month's agent spend and divide it by the number of agent PRs that merged without human rework. If you cannot produce that denominator, that is your first finding.
- Tag your last 50 rejected or reworked agent PRs with Real-SWE's five failure categories. An afternoon with a staff engineer and a spreadsheet gives you your dominant failure mode, which may not match the benchmark's.
This Month:
- Run a private bake-off with enough tasks to mean something — several times Real-SWE's ten, drawn from your own repositories, with verifiers taken from your existing tests, several runs per task, and two or three model-harness pairs. Package it in a reusable format such as Harbor, "a framework from the creators of Terminal-Bench for evaluating and optimizing agents" that supports Claude Code and Codex CLI, so you can rerun the set whenever a vendor ships a new model. Compare models task by task, not on averages.
- Add the one gate check for your dominant failure mode from the mapping above, and track whether your reworked-PR rate falls.
Before Renewal:
- Ask any vendor or benchmark citing private-code results for five things: task count, per-task results, how intervals were computed (clustered by task), harness and reasoning settings, and how cost was estimated, caching included.
- Price the contract on cost per resolved task plus your review break-even, not on seat price or list token rate. On Real-SWE, GPT-5.6 Sol is the second-cheapest model per rollout at $2.65 and only the fifth-cheapest per resolved task at $16.36.
The Bottom Line
Coding-agent selection is repeating the arc of every public benchmark: models and training data converge on the public test until it stops measuring much, and OpenAI said as much about SWE-bench Verified. Private-code benchmarks are the correction, and Real-SWE is a useful one — more for its method than its ranking.
The lasting lesson is that "best model" is the wrong question. The right one has two parts: what does a working change cost, and what does a broken one look like when it lands in your review queue?
The leaderboard tells you who won ten tasks. Your review queue tells you who wins yours.
Continue Reading
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- 221 Green Patches Failed Review. Put Your Rules in Context.
- Claude Matched the Patch. Qwen Overshot. Score the Scope.
- One PR Billed 156M Tokens. Cap the Reads, Not the Rate.
- Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.