Model Router Buyer's Guide: Buy Failover, Not Judgment

Buy the failover. Write the routing logic yourself. A model router is sold as one product but is really two, and only one half currently earns its fee.

The half that works is mechanical: a fallback list, a retry policy, a cooldown on a provider that starts throwing 529s. The half that does not is the learned classifier that promises to read your prompt and pick the cheapest model that can handle it. On RouterArena, the first open leaderboard built to compare routers on the same dataset, the paper's cohort of twelve put a commercial router dead last — not because it was inaccurate, but because it "frequently selects expensive models." A separate benchmark of ten routing methods, LLMRouterBench, found that several published routers — including a commercial one — "fail to reliably outperform a simple baseline": the single strongest model in the pool, chosen with hindsight, handling everything. The best routers in that pool do beat it, by up to 4% accuracy.

So: pay for the traffic plane, keep the decision in your own code, and start with a two-tier cascade rather than a classifier.

Option	What you are actually buying	Router fee at 2M req/mo	Added latency	Model pool
LiteLLM (self-hosted)	Six rule-based strategies plus failover, MIT-licensed	$0 licence; you fund infra and on-call	your own network hop	any provider you configure
Vercel AI Gateway	Pass-through plane with a fallback array	$0 markup, $0 BYOK fee	~20 ms at the edge	hundreds of models
OpenRouter Auto Router	Task classification ranked by community spend	$0 for routing; 5.5% on card credits	not published	full catalogue
Microsoft Foundry model router	Trained difficulty classifier behind one endpoint	~$840 ($0.14 per 1M input tokens)	"negligible fraction" of inference, unquantified	27 models, 5 vendors
Bedrock Intelligent Prompt Routing	Pairwise response-quality prediction	~$2,000 ($1 per 1,000 requests)	~85 ms P90	2 models, same family
Not Diamond	Learned meta-router across providers	~$300 ($0.05 per 1M tokens routed)	100–150 ms	multi-provider; count not published

Fees are the vendors' published list prices, checked 12 September 2026, normalised to one workload: 2 million requests a month at 3,000 input and 500 output tokens each — 6 billion input tokens. They exclude the model tokens themselves, which every option on this list passes through at provider list price. One caveat on the Microsoft row: its documentation says only that model router bills "for input prompts at the rate listed on the pricing page," and the $0.14 rate is taken from a third-party pricing tracker rather than a Microsoft page.

---

A Router and a Gateway Are Not the Same Purchase

A gateway is the traffic plane; a router is the decision function that runs on top of it. The gateway terminates one API, holds the keys, meters spend, retries, and fails over. The router decides which model gets this request. Almost every product in this category sells both, which is why buyers end up paying learned-router prices for what is, in practice, a very good proxy.

The distinction matters because the two halves have completely different evidence behind them. Failover is deterministic plumbing: it either retries on a 500 or it does not, and you can test it in an afternoon. Learned routing is a prediction problem, and predictions have error bars that only show up in your quality metrics eight weeks later.

Keep them separate in your evaluation and in your contract. Buy the plane from whoever gives you the cleanest failover semantics and the least lock-in, and treat the routing policy as application code you own — which is precisely how Vercel's own build-your-own-router guide draws the line: the gateway "provides the mechanics as request-level configuration," while what stays yours is "your routing policy, your keys, your spend rules, and your retention posture."

What the Neutral Benchmarks Say About Learned Routers

Independent evaluation has caught up with this category in the last year, and the results are not kind. Three separate efforts, different teams, same direction.

RouterArena scores routers across 9 domains and 44 categories on accuracy, cost, optimality and latency. Its headline finding is that "commercial routers do not necessarily outperform open-source routers" — commercial entrants bought accuracy by deferring to expensive models, which is exactly the behaviour you are paying a router to avoid. Its live leaderboard now carries 32 entries with cost spanning $0.04 to $10.02 per 1,000 queries, a 250x spread for accuracy that ranges from 32% to 80%.

LLMRouterBench compared ten routing methods against three reference points: a random router, an oracle, and "Best Single" — the model "with the highest average accuracy across all datasets in hindsight." Note that the baseline is itself hindsight-selected, so beating it is a lower bar than it sounds and losing to it is worse than it sounds. Several methods lost to it; the top methods beat it by up to 4%. Notably, the best performer in its cost-performance pool was Avengers-Pro, a clustering approach that requires no neural-network training at all, delivering up to 31.7% cost reduction while matching Best Single accuracy. The paper's own conclusion is that "embedding models have little influence on routing performance" and that capturing "coarse-grained domain structure" drives most of the gain.

The most useful result is the bleakest. "The Routing Plateau" ran 21 routing methods across five benchmarks and found they all converge into a narrow band far below the oracle, caused by what the authors call a predictability bottleneck: "current routers mainly learn global averaged model-performance trends rather than fine-grained query-specific routing signals. As a result, they solve overlapping easy queries but collectively fail on hard queries."

Read that sentence twice before you sign anything. Every router on the market is good at the same easy queries and bad at the same hard ones. The vendor differentiation you are being sold does not exist yet in the measurements.

The Routing Signals That Survive Contact With Production

Two signals hold up, and neither requires a vendor. The first is coarse task clustering; the second is model confidence, used as an escalation trigger rather than a pre-classification.

Pre-routing asks a small classifier to guess difficulty before any model has seen the prompt. It is cheap, it is what almost every managed router does, and it is the failure mode in every post-mortem. Cascading inverts it: send the request to the cheap model first, inspect the answer, and escalate to the strong model when confidence is low. It costs you a wasted cheap call on the hard queries and it is strictly better informed, because it makes the decision with the answer in hand instead of guessing from the question.

The distinction is not academic. The original RouteLLM work trained routers on human preference data and reported cost reductions "by over 2 times in certain cases" without compromising quality, plus transfer learning that survived swapping the underlying strong and weak models. That transfer property is the reason a simple router ages better than a complex one: your model list will change four times before your routing policy needs to.

What does not work as a signal: prompt length, a keyword list, or the user's tier. Length correlates with nothing useful once you have system prompts and tool definitions in the context, and Microsoft's own routing documentation concedes the difficulty directly — in multi-turn conversations "earlier turns provide context but the latest user message carries the most routing-relevant signal," and in long contexts "the routing signal might be distributed across the entire input."

How a 60% Saving Became a Churn Problem

A routing win is measured in the inference bill, and a routing failure is measured in a different budget entirely. That asymmetry is why these regressions run for months.

The clearest public account is a team that built a routing layer and broke their product — written up by Pratik Rupareliya, co-founder of the AI services firm Intuz, about an unnamed client. It is one interested party's reconstruction of an anonymised engagement, not an audited set of books, so weigh the shape of the failure more heavily than the decimal places. They cut the monthly inference bill to 40% of its previous level — roughly $100,000 a month saved — and celebrated at week 8. Quality drift appeared in the regression suite around weeks 3 to 6 and was misattributed to provider drift. Customer satisfaction moved at week 10. Churn crossed baseline at week 13. Attribution took another month. The cost of the degradation was inferred rather than measured, at "conservatively four to five times the cost savings" — $400,000 to $500,000 a month in retention and support.

The failing query was "where is my charge from". It looks trivial. Sometimes it was a fraud investigation.

Three measurement failures let it run, and all three are worth copying into your own design review:

1. Human review was aggregated across tiers. The weighted average of a healthy expensive tier and a sick cheap tier looks fine. Microsoft's evaluation guidance says the same thing from the other direction: "Group prompts by workload category so that an aggregate result doesn't hide a category-specific regression."
2. In-product feedback was too sparse to detect anything — about 3 thumbs-down per 1,000 interactions. That is not a quality signal, it is noise with a UI.
3. The cost moved to a different budget centre. Support headcount absorbed the failure, so nobody in the AI budget meeting saw it.

Their fix was the cascade: start cheap, escalate on low confidence. It gave back roughly 35% cost reduction and held quality. That number — 35%, not 60% — is a more defensible planning figure than the headline, though it is one account rather than a distribution.

Segment Your Evals by Tier, Then Break It Down by Category

Quality regression from routing is invisible in an aggregate score, by construction. If your router sends 70% of traffic to a cheap model and that model is 10 points worse on one category that is 5% of volume, your blended score moves half a point. Your customers notice long before your dashboard does.

Three things make it detectable:

• Log which model served the request, on every request. Both managed routers expose it — Microsoft returns the selection in the model response field and its docs say plainly, "Don't ignore the model field in responses"; Bedrock likewise returns the model that was used. If you build your own, emit it as a span attribute and make it a required dimension on every quality dashboard.
• Score per (category × model), never blended. The pre-merge gate should run a fixed representative set — a few hundred cases is enough if it is stratified — and block any routing change that drops a category, not just the mean. Sizing that set is its own discipline; most teams run far too few cases to distinguish a real regression from sampling noise.
• Change one lever at a time. Microsoft's guidance is explicit: "Keep the workload dataset, baseline, and application configuration fixed when you change the routing mode or model subset," then rerun. If you change the routing mode and the prompt in the same week, you have learned nothing.

And schedule the re-run. A managed router's behaviour changes underneath you: Microsoft's 2025-11-18 version "is updated in place as new models become available," so an endpoint you evaluated in June is routing to a different pool in September without a version bump or a change on your side.

Failover Semantics Are Where the Real Money Is

Failover is the half of this purchase that is worth paying for, and it is the half buyers skim. Get the semantics in writing, because they differ materially between products.

LiteLLM is the most explicit. Its router ships six strategies — simple-shuffle (the recommended default), latency-based-routing, usage-based-routing-v2, least-busy, cost-based-routing, and a custom plug-in base class — plus an order parameter for priority failover, automatic cooldowns after a default of 3 failures per minute for a default 5 seconds, and per-error-type retry policies. That is a complete, inspectable failover contract you can read in an afternoon. It is also all deterministic: there is no learned classifier here, which is the point.

Microsoft Foundry's model router enables automatic failover by default and redirects to "the next most appropriate model" when one misbehaves. The catch is in the custom path: your model subset is your fallback set, so a single-model subset silently disables failover. Microsoft's own best-practice list says "Select at least two models for failover" and warns against using the router as "an expensive passthrough."

Vercel's gateway takes a models array under providerOptions.gateway and executes the fallback chain for you. One detail to price in: with BYOK, when a request on your credentials fails, the gateway retries with system credentials and that fallback usage bills to your credit balance — useful behaviour, but it means BYOK is not an absolute spend boundary.

OpenRouter's Auto Router degrades to a default model set if its own classification or ranking infrastructure fails, so "a request never fails because routing infrastructure hiccuped." That is the right design for a routing layer, and it is the question to ask every other vendor: what happens when your router is down?

The reliability case is also the strongest measured argument for a gateway at all. One independent 12-day measurement of 4,127 assistant turns found time-to-first-token landed between 0.35 and 0.63 seconds on every single gateway trial, while the same provider called directly threw 8- and 9-second outliers on otherwise sub-300ms starts. Tail latency, not median, is what your users feel.

Count the Latency the Router Adds Before the Model Runs

Router latency is pure overhead paid on every request, including the ones that route to the expensive model anyway. It splits cleanly by architecture.

A pass-through plane adds a network hop. Vercel puts its CDN at sub-20ms; self-hosted LiteLLM adds whatever your own hop costs, which is why you co-locate it with the calling service.

A learned router adds an inference. AWS measures Intelligent Prompt Routing at approximately 85 ms at P90. Not Diamond publishes 100–150 ms average added latency — and states its own savings as "at least 20-40%," notably below the headline figures that circulate in this category. Microsoft declines to quantify, describing the routing decision as "a negligible fraction of the LLM inference time," which is true for a 20-second reasoning call and is not true for a 400ms classification step inside an agent loop.

That last case is where routers quietly lose. If your agent makes twelve model calls per task, a 120ms router tax is 1.4 seconds of wall clock per task before a single token is generated — and the cheap-model calls it is optimising are the short ones, where the overhead is proportionally worst. RouterArena flags the same risk: router latency "may even compromise service-level objectives."

Prompt Caching Is the Tax Nobody Prices In

Routing and prompt caching pull in opposite directions, and at agent-scale context the cache usually wins. A cached read is discounted an order of magnitude against fresh input; switching models to save 60% on the per-token rate forfeits the cache entirely and re-pays full price for a context you already paid to write.

Microsoft documents the mechanic without flinching: caching benefits "apply only when the same model handles consecutive requests with overlapping prompt prefixes," and by default the router "repeats this selection process for each request," so separate turns in a stateless conversation may hit different models. Its mitigation is session affinity, currently in preview — and the caveat is the important part. Session affinity "doesn't inspect provider cache signals, compare the benefit of staying with switching, or guarantee cache hits," and the association expires after 30 minutes of inactivity.

So the affinity feature is a heuristic layered on a heuristic. If your workload is long-context and cache-heavy — coding agents, document review, anything carrying a large stable system prompt — compute the cache-adjusted cost before you route. The router may be selling you a discount you are already getting.

Build or Buy: Buy the Plane, Build the Policy

Build the routing policy. Buy the gateway. The asymmetry is that the policy is roughly 200 lines of application code you fully understand, while the plane is provider integrations, key rotation, retries, metering and a schema-compatibility treadmill that Vercel's guide, citing an unnamed gateway vendor, puts at "6 to 12 months of engineering" to replicate.

The economics support it. Managed routing fees are small in isolation and not small at volume. At our 2M-request month, Bedrock's $1 per 1,000 requests is $2,000 — a fee that scales with request count and is indifferent to how many tokens each request carries, which is the worst possible shape for a chatty agent workload. Microsoft's model router bills "for input prompts at the rate listed on the pricing page," with output uncharged; a third-party pricing tracker puts that rate at $0.140 per 1M input tokens, which is $840 on the same workload and scales with context, which punishes exactly the long-context traffic where routing helps least. Not Diamond's $0.05 per 1M tokens routed is the cheapest of the three at roughly $300.

Against that, LiteLLM's open-source proxy is free forever and self-hosted with fallbacks, budgets, virtual keys and Prometheus metrics in the box; the enterprise tier is annual, "sized to your annual gateway request capacity… never per token," and adds SSO/SCIM, audit logs and air-gap deployment. OpenRouter charges nothing for the Auto Router itself and passes provider pricing through "without any markup on inference pricing," taking 5.5% on card credit purchases and 5% on BYOK above a $25,000 monthly allowance. Vercel charges no markup and no BYOK fee, billing add-ons separately — $0.10 per 1,000 requests for team-wide zero-data-retention or a provider allowlist, with the per-request equivalents free.

One shortlist note. Portkey appears in nearly every search result for this category, and its own pricing page now states it ships as the Prisma AIRS AI Gateway under Palo Alto Networks. Palo Alto Networks announced the acquisition on 30 April 2026 and closed it on 29 May 2026, so any review written before mid-2026 describes a different corporate entity; confirm who signs your data processing agreement before you rely on one.

Who Should Not Buy Each of These

The strongest thing a buyer's guide can tell you is where each option fails.

Option	Do not pick it if
Bedrock Intelligent Prompt Routing	You need more than two models, or models from more than one family, or non-English traffic, or routing that learns from your data. All four are documented limits.
Microsoft Foundry model router	You need a deterministic model per request, or your prompts exceed the smallest model in the pool — the effective context window is capped by the weakest member.
Not Diamond	Your workload is agentic and latency-sensitive; 100–150 ms per hop compounds. Also skip it if a last-place finish in RouterArena's published 12-router cohort — 28th of 32 on the live leaderboard as of September 2026 — would be hard to defend in a review.
OpenRouter Auto Router	You need auditable, reproducible routing. Ranking by trailing-7-day community spend means your routing changes when the market's does.
Vercel AI Gateway	You want the vendor to make the model decision. It gives you an excellent plane and a fallback array; the intelligence is your problem.
LiteLLM self-hosted	You have no platform team. "Free" here means you own the uptime of the thing every model call passes through.

Name the loser plainly: Amazon Bedrock's managed router is not a 2026 product. Its documented table of supported models still tops out at Claude 3.5 Sonnet v2, Claude 3.5 Haiku, Amazon Nova Pro and Llama 3.3 70B; the console requires you to "choose exactly two models within the same family"; it is "only optimized for English prompts"; and it states flatly that it "can't adjust routing decisions or responses based on application-specific performance data." In fairness to AWS, the same page markets the feature as "Future-Proof: Incorporates new models as they become available," ties the exactly-two rule to the preview default routers, and gives a CLI example for a configured router that references claude-sonnet-4-5 — so the reachable pool may be newer than the published table. Check your own console before you rule it out. AWS's own headline savings — 56% for the Anthropic family, 35% for Nova, 16% for Meta — are savings within a family, which is the narrowest version of the trade. Paying $1 per 1,000 requests for a pairwise chooser whose published model table stopped at Claude 3.5 is the clearest bad buy in this category.

Second place in that ranking goes to nobody, because the vendor is gone. Martian was on every model-router shortlist two years ago. Its homepage today leads with "Understanding Intelligence," an RL framework for coding agents and an interpretability method — no router, no routing price. That is the base rate for this category, and it should inform how much switching cost you are willing to accept.

Three Things to Do in the Next 30 Days

This week: pull one month of production prompts, strip the secrets, and bucket them by workload category. Score your current single model per category to establish the baseline. You cannot detect a routing regression you have no baseline for, and this is the step every failed rollout skipped.

This month: implement the cascade, not the classifier. Cheap model first, escalate on low confidence, log the serving model and the escalation reason on every request. Run it in shadow against 5% of traffic for two weeks and compare per-category, per-tier — never blended. Compute the cache-adjusted cost, not the list-price cost.

Before you sign anything: get the failover contract in writing. What happens when the router is unavailable? Does a single-model configuration silently disable failover? Does a BYOK failure bill to your account? Does the underlying model pool change without a version bump? Each of those is documented for at least one product on this list, and each of them is a production incident you can price today instead of discovering in November.

The Bottom Line

Every technology cycle produces a layer that promises to make a hard decision for you, prices itself against the savings it claims, and gets measured honestly about two years later. Query optimisers earned it. Autoscalers mostly earned it. Learned model routers have not earned it yet — twenty-one methods converging into a narrow band below the oracle is not a market with a winner in it, it is a market with a research problem in it.

That does not mean do nothing. It means the value is in the parts you can verify: a fallback list that works, a cheap model that handles the easy two-thirds, an eval suite that reports by category, and a log line that says which model answered. All four are cheap. None of them requires a vendor's judgment about your prompt.

Buy the plumbing. The judgment is still yours, and the benchmarks say you are better at it than the router is.

Continue Reading

• Best LLM Gateways for Cost Control: Self-Host First
• Your AI Router Is Trading a 10x Discount for a 2.5x One
• GitHub's Router Bills Every Leg. Find Your Break-Even.
• Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
• Inference Cost per Million Tokens: Price the Task, Not the Rate
• AI Gateway vs API Management: No Managed Gateway Caps Your Spend
• Anthropic Cut Cache Reads 75%. Opus 5 Still Undercuts It.
• Claude vs GPT vs Gemini: Stop Comparing Per-Token Prices