A cheaper model is not a cheaper agent, and there is now a controlled experiment pointing at where the money goes.
Across 90 multi-turn coding-agent trials, Claude Haiku 4.5 consumed 735,000 tokens to build a service that Claude Sonnet 4.6 built with 640,000 — and Haiku scored 1.92 points lower on a 10-point rubric. The mid-tier model spent more to deliver less, because it fell into compilation repair loops the frontier model walked past. If you are about to route your coding agents to a cheap tier to cut spend, the saving is real, but it is not the number on the rate card, it varies fivefold by vendor, and on the weakest models it depends entirely on whether you replaced your prose design docs with something a compiler can check.
What the Experiment Actually Ran
The study is a five-by-six factorial on specification format, not a model leaderboard. Arquimedes Canedo of Siemens Digital Industries Software published "Architecture as Capability Equalizer for Coding Agents" on 22 August 2026, comparing five informationally equivalent ways to describe the same system: informal prose, Mermaid diagrams with numbered constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules.
Every format encoded the same architecture. The task was a task-management API with seven components and 25 HTTP routes, and each agent got up to 50 turns with file-write, file-read, list, shell and done tools. Six models ran three trials per format: Sonnet 4.6, Haiku 4.5, GPT-5, GPT-5-mini, Gemini 2.5 Pro and Gemini 2.5 Flash. A Sonnet 4.6 judge scored architectural adherence, completeness, code quality and constraint compliance, and static import analysis measured route coverage and rule violations independently of the judge.
The headline result is an interaction, not a ranking. On the two frontier models, specification format barely moved quality — a spread of 0.17 to 0.92 points. On the four non-frontier models the spread ran 0.83 to 2.42. Format is a lever whose value scales inversely with model strength — which is the paper's own framing of its result.
The Cheap Tier Saved 62%, Not 67%
Token counts change the downgrade arithmetic, and they change it differently at each vendor. The paper reports tokens, never dollars. Put its token totals against each vendor's published list price and the picture separates cleanly:
| Family | Frontier tier | Cheaper tier | Rate-card saving | Token-adjusted saving | Quality change |
|---|---|---|---|---|---|
| Anthropic | Sonnet 4.6 — 640K tok, 8.42 | Haiku 4.5 — 735K tok, 6.50 | 67% | 62% | −1.92 |
| OpenAI | GPT-5 — 248K tok, 7.08 | GPT-5-mini — 225K tok, 6.72 | 80% | 82% | −0.36 |
| Gemini 2.5 Pro — 329K tok, 6.07 | Gemini 2.5 Flash — 223K tok, 6.35 | ~76% | ~83% | +0.28 |
The right-hand columns are arithmetic on the paper's token totals and each vendor's list price, not figures the paper reports — and the paper reports only combined input-plus-output tokens, so every row assumes the two tiers split their traffic the same way. Two of the three price ratios are exact. Anthropic prices Sonnet 4.6 at $3 per million input and $15 per million output against Haiku 4.5's $1 and $5 — precisely 3.0x on both sides, so the ratio holds whatever the input/output mix. OpenAI prices GPT-5 at $1.25 and $10 against GPT-5-mini's $0.25 and $2 — precisely 5.0x on both. Google's Gemini 2.5 Pro runs $1.25 and $10 against 2.5 Flash's $0.30 and $2.50, a ratio close enough to 4x that the Google row is an estimate rather than an identity.
Read the table as a decision, not a scoreboard. The same move — drop to the cheap tier — costs 0.36 points at OpenAI and 1.92 at Anthropic. A vendor-agnostic "route everything cheap" policy is not one policy. It is three, and one of them is bad.
This is the mechanism Janakiram MSV described in Forbes on 13 July 2026, arguing that total spend is task volume times attempts per task times tokens per attempt times price — four variables that move independently, so a falling unit price can coexist with a rising bill. Canedo's contribution is a measured value for the middle term. We have made the same point about comparing frontier models on per-token price; this is the first controlled number behind it.
Low Token Counts Can Mean the Agent Stopped Checking
The most efficient-looking model in the study was the one that never tested its own work. This is the finding that should change what your dashboard measures.
Sonnet 4.6 ran the demo end-to-end in 100% of trials. Haiku 4.5 managed 80%, GPT-5 53%, GPT-5-mini 40%, Gemini 2.5 Pro 20% — and Gemini 2.5 Flash never ran it once, across any trial. Compile attempts track the same gradient: Haiku invoked the TypeScript compiler 8.6 times per trial and Sonnet 5.1, while Gemini Flash managed 2.1.
Now line that up against the token bill. The two models that validated most are the two that spent most. The model that validated never is among the three cheapest. Haiku's 735,000 tokens are not waste in the way an idle GPU is waste — they are a model trying to make the build pass and taking eight attempts to get there. Flash's 223,000 are cheap because it declared victory without running anything.
GPT-5 is the honest counterexample: it validated in 53% of trials on 248,000 tokens, so convergence speed matters as much as diligence, and the correlation is not clean across all six. But the extremes are unambiguous, and the operational lesson survives: a tokens-per-task metric silently rewards the model that skips the tests. If your FinOps review ranks coding agents by spend, it will keep promoting the one that ships fastest by not looking. That is the same trap as grading agent pilots on an average score instead of a completion rate, and it is why repair loops need a hard iteration budget rather than an open-ended retry.
Gemini Flash Shipped One-Third of Your API
Deterministic measurement caught a failure the LLM judge nearly missed. Under prose, Mermaid and OpenAPI specifications, Gemini 2.5 Flash implemented 33% of the specified routes; under C4 it reached 67%, and under TypeScript interface contracts it reached 100%. On 25 routes, that is roughly eight endpoints delivered against 25 asked for — and the judge still gave the prose run 5.75 out of 10.
Sonnet 4.6, Haiku 4.5 and GPT-5 hit 100% route coverage on every format. The only other misses were GPT-5-mini at 85% under prose and Gemini 2.5 Pro at 67% under C4 — every gap lands in a non-frontier tier, which is exactly where a cost-cutting mandate pushes work.
Two things follow. First, static analysis found what the rubric blurred: a judge scoring "completeness" as one dimension among four will not reliably fail an API that is two-thirds absent, and the paper's own judge-versus-automation correlation for completeness is only r=0.63 for the Sonnet judge that scored all 90 trials. Second, TypeScript contracts are the only format in the study where all six models hit 100% on both route coverage and constraint compliance. That unanimity comes from a census, not a sample, which makes it stronger evidence than the judge scores sitting next to it.
Be careful about generalising it into "always use TypeScript contracts." The judge scores say otherwise in one place that matters: TypeScript contracts were Gemini 2.5 Pro's worst format at 4.50 against OpenAPI's 6.92, and GPT-5's best at 7.50 while OpenAPI was its worst at 6.58. There is no format that wins everywhere on quality. There is one that never leaves endpoints on the floor.
On Frontier Models, the Diagrams Are the Waste
If you run frontier models, the spec-format debate is costing you money in the opposite direction. Prose gave Sonnet 4.6 an 8.42 at 433,000 tokens; Mermaid with constraints and ADRs gave it 8.50 at 1,060,000. That is 2.4x the token bill for eight-hundredths of a point. At Sonnet 4.6's published rates the 627,000 extra tokens are worth between $1.88 and $9.41 per trial depending on the input/output split — per service, on one seven-component system, before you multiply by a team.
Both figures are means of three runs, and the runs are wild. Prose consumed 367K, 602K and 332K tokens; Mermaid consumed 1,735K, 412K and 1,032K. One Mermaid trial came in cheaper than the prose average. The gap between the means is large enough to be worth chasing on your own repository and nowhere near clean enough to bank.
And then the finding nobody will put on a slide. With no architecture specification at all, Sonnet scored 5.08 against 8.42 with plain prose — architectural adherence 3.00 versus 7.67, constraint compliance 3.00 versus 8.00. Writing any design document is worth 3.34 points on a frontier model. Choosing the right format is worth 0.17. The argument your architects are having about C4 versus Mermaid is a twentieth of the size of the argument they should be having about whether a spec gets written at all.
What This Study Cannot Tell You
The design is underpowered and the author says so. With n=3 per cell across a 5x6 grid, the paper reports no statistical significance tests, uses one system of moderate complexity, no open-weight models, specifications written by the author, and Sonnet 4.6 as judge for Sonnet-generated code. Treat every judge score as directional — and treat the token counts the same way, including the headline. Sonnet's three prose trials consumed 367K, 602K and 332K tokens, and its three Mermaid trials 1,735K, 412K and 1,032K. Against a spread like that, a 15% gap between two model-level means is a reason to go and measure, not a measurement. What survives the noise is route coverage and constraint compliance: those are censuses of the generated code, not averages of three runs.
The model set is also drifting out from under it. Haiku 4.5 is current, but Sonnet 4.6 is not, and the economics have moved since the trials. Claude Sonnet 5's introductory $2 and $10 per million tokens is now its standard price, and the increase that had been scheduled for 1 September 2026 will not happen — 2x Haiku's rate rather than 3x, which on the paper's raw token counts would cut the modelled Anthropic saving from 62% to roughly 43%. That arithmetic does not survive the same page's other note: Claude 4.7 and later use a tokenizer emitting about 30% more tokens for the same text, and Sonnet 5 is on it while Haiku 4.5 is not. Inflate the frontier side by that much and the saving comes back to roughly 56%. Two modelled numbers thirteen points apart, off one pricing page, is the argument for measuring your own traffic rather than trusting either. On the OpenAI side the GPT-5.6 family now sits above the tested pair, though GPT-5 and GPT-5-mini are still listed at the prices used here.
Nor does the study measure security. BaxBench, a 2025 benchmark of 392 backend-generation tasks, found 62% of solutions from even the best model were incorrect or contained a vulnerability, with roughly half of functionally correct solutions carrying a security flaw. A spec format that gets all 25 routes implemented has not established that any of them are safe.
What to Do Before You Downgrade a Tier
This Week:
- Change the denominator on your token metric. Report tokens per merged pull request, not tokens per run. Haiku's 15% token premium is invisible in a per-call chart and obvious in a per-outcome one.
- Log whether the agent runs anything. Add one counter: the share of agent sessions that executed a build, test or run command before declaring the task done. If you cannot produce that number today, you cannot tell diligence from a cheap dashboard.
- Diff routes against the spec on your last ten agent-built services. Count implemented endpoints versus specified ones. You are looking for the Flash failure — silently partial, not visibly broken.
This Month:
- Run the A/B on your own repository before the downgrade, not after. Same ticket, same specification, both tiers, three runs each, and measure tokens-to-green plus route coverage. The paper had n=3 on somebody else's codebase; you can beat that on yours in an afternoon.
- Convert one prose design doc to machine-checkable form. OpenAPI describes the HTTP surface in YAML or JSON as a language-agnostic contract; typed interfaces pin the service boundaries; and a rules file the build can fail on pins the rest — ArchUnit for the JVM checks package dependencies, layers and cyclic dependencies from an ordinary unit test, with equivalents in the TypeScript and Python ecosystems.
- If you are on a frontier tier, meter what a diagram-heavy spec costs the agent before you commission the next one. Human readers may still want the C4 diagram. In this study the agent charged 2.4x for it on average and returned 0.08 of a point — across three noisy runs, which is enough to justify measuring your own and not enough to justify a policy.
Before Renewal:
- Re-derive any commitment sized on rate-card ratios. If a usage commitment assumed a tier downgrade saves the sticker percentage, the token-adjusted number is lower at Anthropic and higher at OpenAI, and the Sonnet 5 repricing moved the goalposts again. Bring measured tokens-to-green, not a price sheet.
The Bottom Line
Every previous cycle of "just use the cheaper engine" ended the same way: the saving showed up on the unit price and the cost reappeared somewhere nobody was instrumented to see. Offshore development moved the hourly rate and relocated the expense into rework. Serverless moved the server bill and relocated it into cold starts and egress. This one relocates it into repair loops and unimplemented endpoints — and, in Gemini Flash's case, into a service that looks finished, scores 5.75, and is missing two-thirds of its API.
The fix is not to stay on the frontier tier. GPT-5-mini took an 82% saving for a third of a point, which is an excellent trade. The fix is to stop buying the tier and start buying the outcome, and to give the cheap model something a machine can grade it against before you ask it to work unsupervised. Prose is a specification your compiler cannot read. So is a diagram.
Downgrade the model if the numbers say so. Do not downgrade the contract.
Continue Reading
- Claude vs GPT vs Gemini: Stop Comparing Per-Token Prices
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
- Your Terraform Repair Loop Broke Passing Checks. Stop at 3.
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
- Best AI Coding Assistant at 500 Seats: Buy Copilot Business