Your inference bill has two discounts available, and you cannot claim both. The first is the list-price spread between a frontier model and a cheaper one — the number on slide three of every multi-model routing pitch. The second is prompt caching, which bills a repeated prompt prefix at 10% of the base input price on the Claude API and at exactly 10% of standard input across the GPT-5.6 family. The cache discount is bigger. A router that switches models mid-session forfeits it on every hop, and in agentic workloads that trade loses money on arithmetic you can check with a calculator.
This is not an argument against having a gateway. It is an argument that the thing your gateway is being sold to do — pick a cheaper model per request — is the one function that has stopped paying for itself.
What Manifest Shut Down After Four Months in Production
A team running a four-tier model router across four vendors killed it and published the reasoning. On 31 July 2026, Bruno Perez of Manifest wrote up why the company deprecated its LLM router: a classifier that sorted every request into "simple, standard, complex and reasoning" tiers and dispatched it to Anthropic, OpenAI, DeepSeek or Mistral. It launched in March 2026, served 7,000 cloud users, and was deprecated in June, with final shutdown set for 1 September. "After four months of usage across 7000 cloud users, we saw mixed results," Perez wrote.
Three failures did it, and only one is about money. The first is classification: "The prompt alone does not contain the whole task; it is just the trigger." A three-word instruction can open a two-hour refactor. The second is coherence — "Jumping from a model to another during working sessions results in lower quality of the overall work." The third is the cache. Perez's own figure is that "cache reads are between 75% and 90% cheaper than uncached inputs," and effective caching requires sticking to one model, which is precisely what a router refuses to do. His closing line is the one to bring to your next vendor call: "The amount saved is paid somewhere else, and that cost is harder to estimate."
The post reached 129 points on Hacker News, where the response split along a useful line. The commenter maxrev17 reported a year of implementation ending in the same place — task-specific model selection, no router. owenthejumper argued routing belongs on the client side. But coffinbirth pointed out that routing services carry GDPR compliance, ISO certification and guaranteed zero-data-retention for regulated buyers, and try-working reported achieving cache hit rates above 99% while still routing between GPT and DeepSeek — by keeping the model pool small and the differences sharp. Hold onto that last one. It is the design that survives.
The Arithmetic That Kills the Per-Request Router
Route a step to a model 2.5x cheaper on list price and the step gets roughly 3.5x more expensive, because you traded a 10x discount for a 2.5x one. The numbers below use published list prices as of 2 August 2026 and a real workload shape rather than a hypothetical one.
The University of Washington's TraceLab study, released 29 June 2026, traced roughly 4,300 coding-agent sessions containing about 350,000 LLM steps and 430,000 tool calls — the researchers' own day-to-day use of Claude Code and Codex, contributed by 43 developers over roughly eight months. That is a narrow and self-selected population, so read the medians as one heavily instrumented workload shape rather than an industry average. Its median Claude step reads 126,180 prefix tokens, appends 857 new ones, and emits 252 output tokens. That is an input-to-output ratio of roughly 500:1, and it means 99.3% of the input tokens on an agent step are context you already sent.
| Per agent step (TraceLab median) | Stay on Claude Opus 5, warm cache | Route to Claude Sonnet 5, cold cache |
|---|---|---|
| 126,180 prefix tokens | $0.0631 (cache read, $0.50/MTok) | — |
| 127,037 input tokens | — | $0.2541 (base input, $2/MTok) |
| 857 appended tokens | $0.0043 ($5/MTok) | included above |
| 252 output tokens | $0.0063 ($25/MTok) | $0.0025 ($10/MTok) |
| Step cost | $0.0737 | $0.2566 |
Sonnet 5 is 2.5x cheaper than Opus 5 on input and half the price on output. Routing to it costs 3.5x more. That $2/MTok is introductory pricing through 31 August 2026; from 1 September Sonnet 5 lists at $3/MTok input and $15/MTok output, which narrows the list-price spread to 1.67x and widens the routing penalty to 5.2x. And that table is the charitable version — if your router also caches on the target model, you pay the 1.25x write multiplier instead of base input, and the step lands at $0.32.
The break-even is a single rule you can put in a design doc. Staying costs the prefix at 0.1x. Routing costs it at 1.0x. So a downstream model only wins if its base input price is more than 10x cheaper than the model you left. From Opus 5 at $5/MTok, the target has to be under $0.50/MTok. Claude Haiku 4.5 at $1/MTok does not qualify — routing there still costs $0.128 per step, 1.7x more than staying put. GPT-5.6 Luna at $0.20/MTok does qualify, at 25x. That is the whole viable routing set: not "cheaper," but an order of magnitude cheaper, and only if you then stay there long enough to warm the new cache.
Why the Cache Is Prefix-Bound and Model-Bound
A prompt cache is a hash of an exact prefix held by one model, which is why crossing a model boundary is always a guaranteed miss. The vendor documentation is unambiguous on both halves of that sentence.
Anthropic's prompt caching guide hashes the entire prompt prefix up to a breakpoint and requires 100% identical prefixes for a hit, allows up to four breakpoints, and enforces a strict tools → system → messages hierarchy: change your tool definitions and the entire cache is invalidated, not just the tools block. Caches are isolated per workspace on the Claude API, AWS and Microsoft Foundry, and per organization on Bedrock and Google Cloud. OpenAI's prompt caching documentation is the same shape: automatic above 1,024 tokens, "cache hits are only possible for exact prefix matches within a prompt," not shared between organizations, and a 30-minute retention that is the only supported value on GPT-5.6, and also the default. Google's context caching sets its minimum per model — 2,048 tokens on Gemini 2.5 Flash and Pro, 4,096 on Gemini 3.5 Flash — and tells you to put large, common content at the beginning of the prompt.
Three vendors, three implementations, one property: the cache belongs to a model, and it is addressed by a byte-identical prefix. There is no cross-model cache and there will not be one, because the artifact being cached is that model's own key-value state. Routing does not merely fail to benefit from caching. Routing is the operation that breaks it.
Why Routing Worked on Benchmarks and Fails in Agents
Routing was validated on stateless single-turn chat, and agentic workloads are the exact inverse of that. The canonical result is RouteLLM, published by LMSYS on 1 July 2024, which reported over 85% cost reduction on MT Bench while retaining 95% of GPT-4's performance, plus 45% on MMLU and 35% on GSM8K, routing between GPT-4 Turbo and Mixtral 8x7B. The accompanying paper shows the best router sending only 14% of calls to GPT-4. That work is sound and its numbers hold — for the workload it measured.
MT Bench prompts are short, independent and single-turn. There is no reusable prefix, so there is no cache discount to forfeit, so list-price arbitrage is the only lever and routing captures all of it. Now look at what TraceLab measured in an agent: sessions averaging 9.2 requests, 8.9 user-initiated steps and 73.6 tool-initiated steps, each one re-sending a context that has grown all session. Measured prefix cache hit rates come in at 95.8% for Claude and 95.7% for Codex overall — and the split underneath is the interesting part. Tool-result steps hit 97.9% and 97.2%. User-initiated steps hit only 86.9% and 78.2%, because humans pause: TraceLab found that once the gap exceeds five minutes low-hit-rate steps appear, and "after 1 hour, almost all steps miss the cache."
That is why Anthropic sells a 1-hour TTL at a 2x write multiplier instead of the standard 1.25x. It is not an upsell. It is a hedge against your user going to lunch, and on a workload where the prefix is 126,000 tokens it pays for itself the second time it is read. Our earlier coverage of agentic token costs running 12x over CIO expectations and of Uber's budget burned in four months both described this overrun without naming the mechanism. The mechanism is prefix re-transmission, and the fix is cache locality, not a cheaper model.
The Strongest Case for Keeping the Gateway
Everything a gateway does except per-request cost routing still earns its place, and some of it is not replaceable in-house. OpenRouter raised $113 million on 26 May 2026 in a round led by CapitalG, and disclosed serving 25 trillion tokens per week — a fivefold increase from 5 trillion six months earlier — across 8 million users. Co-founder and CEO Alex Atallah put the counter-argument as directly as anyone has: "Running inference at scale is fundamentally a multimodel problem. The era of picking a single model is over. Success now depends on continuously routing across a changing market."
He is right about the market and, on the evidence above, wrong about the cadence. Continuous routing across a changing market is a procurement discipline measured in weeks. It is not a dispatch discipline measured in milliseconds. A gateway gives you failover when a provider degrades, one normalized API instead of four SDKs, per-team spend caps, an audit trail, and the compliance envelope that HN commenter coffinbirth named — the reason a regulated buyer signs at all. None of that requires changing models inside a live session, and we made the vendor-concentration case for multi-model architecture on exactly those grounds.
The honest scope statement: list-price routing still wins on short, stateless, high-volume work. Classification, extraction, moderation, embedding-adjacent scoring, tier-1 triage on a 400-token prompt — no reusable prefix, no cache to lose, and the 30-85% savings that our earlier routing coverage and the OpenRouter analysis described are real there. That arithmetic has not gone stale; its domain has just turned out to be narrower than the pitch. What changed in 2026 is that the expensive half of enterprise inference stopped being stateless chat and became long-running agents, and our multi-model default piece priced that half off the wrong lever.
The Number Your Cost Dashboard Does Not Show
Cache hit rate is the single largest cost variable in an agentic stack, and almost no FinOps dashboard reports it. As engineer Tian Pan put it in an April 2026 write-up, "cache hit rate is the most impactful cost lever in an LLM stack, and most teams treat it as a one-time setup task rather than a production metric."
Two production numbers show the range. Deriv published results for its Amy assistant on 23 July 2026: 85.8% prompt-prefix reuse in production and a 77% reduction in input-token costs, against roughly 20% for a poorly structured implementation. They got there by separating static instructions from dynamic context, moving shared dynamic context ahead of conversation history, and rendering prompt blocks deterministically — down to sorting skills by name before rendering. Their warning is the operational one: "A timestamp, reordered list, inconsistent whitespace, or block rendered in a different sequence can break an otherwise reusable prefix."
The second is sharper. ProjectDiscovery's Neo platform sent a static system prompt of 2,500-plus lines of YAML — over 20,000 tokens per agent — on every request, but interleaved working memory, skills and runtime context into the middle of it. Moving that dynamic content out to the conversation tail took the hit rate from 7% to 84% and cut overall cost 59%, on figures the company says come from actual reported costs rather than estimated pricing. It is their own platform and their own number, so read it as a best case — but the fix was prompt assembly and nothing else. No model was changed and no vendor was renegotiated.
That is the comparison worth sitting with. Your router is fighting for a 2.5x list-price spread and losing. A team that moves its per-request fields below the cache breakpoint can take out more than half the bill without opening a model contract. We have written before about how few teams can even see what their agents cost; cache hit rate is the specific missing gauge, and unlike most FinOps metrics it comes back in the API response body for free — Anthropic returns cache_read_input_tokens and cache_creation_input_tokens on every call.
Fix the Cache Before You Fix the Router
Every item below costs less to execute than a routing evaluation and shows up in a single billing cycle. Do them in order; the first three are a week of one engineer's time and will tell you whether the rest of the list is even worth opening.
This Week:
- Instrument cache hit rate as a first-class metric. Pull
cache_read_input_tokensagainst total input tokens per model, per service, per day, and put it on the same dashboard as spend. If you cannot produce that ratio for last week, you do not know what your inference costs. - Run the break-even on your own two models. Take your current model's base input price, divide by ten, and ask which candidate models sit below that line. If none do, per-request cost routing cannot pay for itself in your stack and the analysis is finished.
- Grep your prompt templates for anything that changes per request above the cache breakpoint — timestamps, session IDs, user names, unsorted lists, non-deterministic map iteration. Each one is a total prefix invalidation, not a partial one.
This Month:
- Move routing from per-request to per-session. Choose the model at session start on task class, then pin it. This is the design the HN commenter reporting 99%-plus hit rates converged on: a small, sharply differentiated pool with sticky assignment. You keep vendor optionality; you stop paying for it every turn.
- Freeze your tool definitions behind a release process. Anthropic's invalidation table puts tools at the top of the hierarchy, so a one-word edit to a tool description invalidates the entire cache for every session using it. Treat tool schemas like a database migration, not like config.
- Price the 1-hour TTL against your actual idle-gap distribution. If your assistant is human-paced, gaps over five minutes are your dominant miss cause. The 2x write multiplier is cheap insurance at a 126,000-token prefix; at a 3,000-token one it is not. Measure before you enable it.
Before Your Next Gateway Renewal:
- Separate the two things you are buying. Failover, API normalization, spend controls, audit and the compliance envelope are worth paying for. Automatic per-request model selection is a feature to turn off, not a reason to sign. Ask the vendor for cache hit rate broken out by model on your own traffic — if they cannot report it, they cannot manage the lever that dominates your bill.
- Re-baseline the savings claim. Any routing ROI model built on list-price spreads and no cache term is measuring the wrong quantity. Ask for the number recomputed with cache reads at 0.1x, and watch what happens to the payback period.
The Bottom Line
We have been here before. The industry spent the late 2000s building database query routers that shipped reads to the cheapest replica, and then rediscovered that connection pooling and a warm buffer cache mattered more than which box answered — because the cost of a query was never the CPU, it was the cold page. Prompt caching is the buffer cache of inference, and the model is the box. Routing optimizes the wrong layer with real precision.
None of this means committing to one vendor. GPT-5.6's price cut, Gemini 3.6 Flash's agent token economics and Shopify's distillation work all show the market repricing every few weeks, and you should keep the ability to move. Move at the session boundary, at the service boundary, at the quarterly review — anywhere except the middle of a running agent. And when your CFO asks the question most of them are now asking, the answer is no longer cost per token. It is what fraction of your context you are paying full price to send twice. If you are choosing an orchestration layer at the same time, the framework decision should be judged on whether it preserves a stable prefix across a session, which is a question nobody currently asks in an RFP.
Route your vendors. Pin your models. The cheapest token is the one you already sent.
Continue Reading
- Still Using One AI Provider? You're Overpaying by 67%
- OpenRouter's $113M Bet: Intelligent Routing Slashes AI Inference Bills 30-85%
- Agentic AI Costs 12x More Than CIOs Expected
- AI Agents Doubled. Only 26% Know What They Cost.
- Your AI Vendor Just Became Your Biggest Risk
- LangGraph vs CrewAI vs AutoGen: One of Them Is Retired