The harness moved the score more than the model did. In a 176-setting controlled study of coding-agent harness design submitted on 17 September, changing one context-management setting swung a single model from 6.40% to 58.40% on SWE-Bench Verified — 52 points, same model, same prompt, same tasks. The entire spread between the best and worst model in that study, at a generous context budget, was 43 points. If your coding-agent bake-off held the harness fixed and varied the model, you did not measure the models. You measured how well your scaffold happens to fit each one.
The Harness Swing Was Bigger Than the Model Gap
A coding-agent harness is the software wrapped around the model: the tool definitions it may call, the planning scaffold that makes it keep a task list, the context manager that decides what falls out of the window when the transcript outgrows it, and the loop that ties the three together. Most evaluations treat that bundle as one object. This one does not — it fixes the execution loop and varies three components independently across 176 matched settings: five context-management strategies, four context-window budgets (32k, 64k, 96k, 128k), and targeted planning and action-space ablations, for four models on SWE-Bench Verified (500 tasks) and Terminal-Bench 2.1 (89 tasks).
Read the results by column and they look ordinary. Read them by row and the point lands. At a 32k budget with context management switched off, Nemotron-3 550B resolves 6.40% of SWE-Bench Verified — below the 9.40% managed by Nemotron-3 30B, a model roughly eighteen times smaller. Turn summarization on, hold everything else, and the same 550B at the same 32k budget scores 58.40%.
The mechanism is unglamorous. With no context management at 32k, 78.7% of SWE-Bench runs and 61.0% of Terminal-Bench runs die of context overflow; at 128k that falls to 8.7% and 12.1%. Every managed tier records zero overflow failures at every budget. Bigger models do write longer trajectories — the 550B's median run is 70 to 74 turns against the 30B's 39 to 42 — but the paper reports overflow only as a model-averaged rate, and offers no explanation for why its largest model fares worst at 32k. Most of what context management buys is not better reasoning — it is not dying mid-task.
That benefit decays as the budget loosens, and the study quantifies the decay: the managed-versus-unmanaged gap narrows from 35.7 points at 32k to 15.9, 5.5 and 2.7 points at 128k on SWE-Bench, and from 9.5 to 2.8 on Terminal-Bench. Above roughly 100k, context management stops being an accuracy lever and becomes a cost lever. That matches what OpenHands reported when it shipped its condenser: per-turn API cost fell to under half while resolve rate moved from 53% to 54% — a rounding error on accuracy, a halving on the bill. We have priced that context tax on real pull requests before, and it is the largest line item most engineering orgs are not looking at.
One Flag Decided Which Model Won
The cleanest result in the paper is the action-space ablation, because it is a single binary flag: give the agent a predefined tool set (read_file, write_file, edit_file, list_files, glob_files, grep_text, web_fetch, bash) or strip all of it and leave bare bash. Everything else — loop, planning, context management, budget — is held constant at the most generous setting.
| Model | SWE-Bench, full tools | SWE-Bench, bash-only | Cost, full tools | Cost, bash-only |
|---|---|---|---|---|
| Nemotron-3 30B | 25.20% | 10.20% | $0.09 | $0.03 |
| Nemotron-3 120B | 44.00% | 42.40% | $0.34 | $0.35 |
| Nemotron-3 550B | 65.80% | 69.40% | $2.33 | $1.11 |
| Mistral-Medium-3.5-128B | 68.60% | 45.40% | $3.14 | $1.72 |
Look at the bottom two rows as a procurement decision. With the full tool set, Mistral-Medium-3.5 beats Nemotron-3 550B by 2.80 points and costs 35% more per task. Strip the tools and leave bash, and the 550B beats Mistral by 24.00 points while costing 35% less. Same benchmark, same 500 tasks, same context budget, one flag. The winner reverses and the margin grows eightfold.
It is worse than a per-model effect: it is a per-model-and-per-task-type effect. That same bash-only flag costs Mistral-Medium-3.5 23.20 points on SWE-Bench and gains it 6.74 points on Terminal-Bench (37.08% to 43.82%). One setting, one model, opposite signs on two benchmarks. The authors' own conclusion is that harness design is "a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default."
This is the same shape as the UNICEF finding that a generic MCP connector scored worse than giving the agent no tools at all. Tool surface is not a monotonic good. More affordances help a model that cannot compose a shell pipeline and get in the way of one that can.
Planning Became a Cost Lever, Not an Accuracy Feature
Planning — a persistent task list the model maintains through an update_plan tool — changes role depending on model strength, and the direction is not the one most teams assume.
For Nemotron-3 30B, turning planning off drops SWE-Bench success from 25.20% to 13.60% and Terminal-Bench from 13.48% to 8.99%. It is load-bearing. For Nemotron-3 550B, turning planning off raises accuracy from 65.80% to 67.80% — while raising cost from $2.33 to $3.31 per task. For Mistral-Medium-3.5, planning costs 0.40 points of accuracy and saves 32% of the bill ($3.14 versus $4.65). On the strong models, planning is a budget control that you pay for in a couple of points of accuracy. On the weak one, it is the difference between a usable agent and a broken one. It also changes how long agents run: planning increased Nemotron-3 30B's median turn count on SWE-Bench by 293.2%.
The same section contains the most useful negative result in the paper. One tier made elided content recoverable — the agent could call recall_event to pull back a discarded observation. 56.3% of the settings offering it never invoked it once, and the median invocation rate was zero across all configurations. The heaviest user of recall scored 3.37 points below the simpler tier that just threw the content away. Lossless recall was machinery the models did not use and did not benefit from.
That is worth holding next to what the vendors ship. Anthropic's context editing API exposes trigger (default 100,000 input tokens), keep (default 3 tool uses), clear_at_least, exclude_tools and clear_tool_inputs — real knobs, with no published accuracy numbers attached to any setting. OpenHands exposes max_size and keep_first on its summarizing condenser. Those defaults were chosen by someone, for some model, against some task mix. None of them is your model, and none of them is your task mix.
The Public Leaderboard Already Shows the Flip
You do not need a preprint to see this effect; it is sitting in a public leaderboard. The Terminal-Bench 2.1 release notes publish agent-model pairs on the same 89 tasks, and the same model scores very differently depending on the harness driving it.
GPT-5.4 scores 77.3% under Codex CLI and 54.8% under Terminus 2 — a 22.5-point spread from the scaffold alone. GPT-5.4 mini spreads wider still: 66.1% under Codex CLI, 36.9% under Terminus 2. Opus 4.6 scores 70.1% under Claude Code and 63.8% under Terminus 2. Sonnet 4.6, 58.5% and 51.5%.
Now run the bake-off two ways, as a buyer actually would. Hold the harness fixed at Terminus 2 — the matched-harness protocol every serious evaluation team believes is the rigorous one — and Gemini 3.1 Pro wins at 70.7%, beating GPT-5.4's 54.8% by 15.9 points. Let each model run in its vendor's own CLI instead, and GPT-5.4's 77.3% beats Gemini 3.1 Pro's 67.1% by 10.2 points. The order reverses, and across the full five-pair table Gemini 3.1 Pro falls from first to third. The protocol you chose, not the models, decided the answer.
And the obvious heuristic does not save you: Gemini 3.1 Pro scores higher under Terminus 2 (70.7%) than under Gemini CLI (67.1%). "Use the vendor's own harness" is not a rule either. Neither is "use the one that topped the leaderboard" — one agent-model pair topping a table tells you nothing about the model you are actually buying.
What This Study Does Not Prove
Four limits, stated plainly, because they change how much weight the result carries.
The models are not the ones you are choosing between. The ablation runs Nemotron-3 at 30B, 120B and 550B plus Mistral-Medium-3.5-128B. Those are a clean capability ladder for a research question and not the frontier models on most enterprise shortlists. The direction of the findings — weak models need tool scaffolding, strong models need budget control — is a claim about capability, and it should transfer. The specific crossover point should not be assumed.
The leaderboard comparison is not an ablation. Codex CLI and Terminus 2 differ in dozens of ways at once: prompts, tool schemas, retry policy, compaction. A 22.5-point gap tells you the harness matters; it does not tell you which component caused it. That confound is exactly what the controlled study exists to remove, and why the two pieces of evidence are worth more together than either alone.
Some of these gaps are noise. Terminal-Bench 2.1 is 89 tasks. A 6.3-point difference is about 5.6 tasks — comfortably inside the range where most apparent model gaps turn out not to be real. A 22.5-point difference is about 20 tasks, and that one is real. Apply the same discipline to your own numbers before you re-platform on them.
A comparable study found a far smaller accuracy effect. In June, Vats and Golev ran two models across three open-source harnesses — Goose, OpenCode and OpenHands-SDK — on a 50-task Terminal-Bench Pro subset, and measured paired within-model pass-rate differences of only 0 to 8 points, with bootstrap confidence intervals covering zero for every gap but the largest. Token cost per solved task moved up to 40x. The two results reconcile: comparing three competent harnesses is not the same experiment as ablating one component down to a degenerate setting, and a swing that needs a broken configuration to appear is not one you will meet in production. Both studies agree on the part that held — the cost effect is the large and reliable one. Weigh price per resolved task first, ranking second.
The strongest counter-argument deserves stating: if a vendor tunes its own CLI to its own model, that pairing is a product you can simply buy, and the harness-model interaction is someone else's problem. That is true right up to the moment you want to switch models, run two vendors behind one gateway, or self-host — at which point you inherit the tuning job, and the harness's error handling and retry behaviour become yours to get right.
Fix the Bake-Off Before the Next Renewal
This Week:
- Write down your harness configuration as a versioned artifact — tool list, planning on/off, compaction strategy and threshold, context budget, retry policy. If nobody can produce that document, your last bake-off is not reproducible and its result is not evidence.
- Re-run your two finalist models with the file tools removed and bash left in place. It is a config change, not a project. If the ranking moves, you have the paper's result in your own codebase and your shortlist is wrong.
- Pull per-task cost, not just success rate, out of your eval harness. Across the three ablations of Nemotron-3 550B at a 128k budget, cost per task ranged from $1.11 to $3.31 — a 3x spread — while success rate moved only between 65.80% and 69.40%. The cheapest configuration was also the most accurate one. Accuracy-only scorecards hide that entirely.
This Month:
- Test at your real context budget, not the maximum. Most of context management's accuracy value appears below 64k and is nearly gone by 128k. Which knob matters depends on the budget you actually run in production.
- Delete the harness machinery your agents do not use. Instrument tool-call frequency for a week; anything with a median invocation rate of zero is cost and attack surface with no measured return.
- Put a harness re-tune on the critical path of every model swap, with an owner and an estimate. A model switch is not a config-file edit. The scaffold that made model A cheapest can make model B both slower and more expensive, and cost per resolved task on your own code is the number that decides it.
Before Renewal:
- Demand the harness configuration behind any benchmark number a vendor quotes you. A resolve rate without a scaffold specification is not a measurement. If they will not provide it, treat the number as marketing.
- Add a re-benchmarking trigger to the contract for silent model updates, and run the evaluation on both the matched harness and the vendor's native one — the two rounds can disagree, and both results are information.
The Bottom Line
Every generation of infrastructure buying has a moment where the industry discovers it was benchmarking the wrapper. Databases had it with connection pooling and query planners. Virtualization had it with hypervisor and guest-driver tuning, where the workload you thought you were measuring turned out to be settings nobody in procurement had heard of. Coding agents are having it now, and the evidence arrived in the same order: practitioners noticed their numbers would not reproduce, the leaderboards showed the same model scoring twenty points apart, and only then did somebody run the matched ablation that says which component is responsible.
The practical consequence is narrow and expensive. A model swap is not a drop-in, and the discount you negotiated on tokens can be erased by a scaffold that was tuned for the model you are leaving. Budget the re-tune, or you will pay for it anyway — in a bill nobody forecast and a benchmark nobody can reproduce.
You did not run a model bake-off. You ran a fit test, and you never wrote down the fixture.
Continue Reading
- What Is an Agent Harness? The $1B Claude Code Architecture
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- Fable Cost $18 a Fix on Private Code. Flash Cost $8.
- One PR Billed 156M Tokens. Cap the Reads, Not the Rate.
- UNICEF Tested a Generic MCP Server. It Lost to No Tools.
- Claude Matched the Patch. Qwen Overshot. Score the Scope.
- Wiping the Transcript Was the Worst Retry. Paraphrase.
- AI Agent Harness Pricing: Hidden Infrastructure Costs