NVIDIA NeMo Switchyard
by NVIDIA
Open-source model router that sends each agent step to the cheapest model that can still do the job
NeMo Switchyard is an Apache 2.0 model-routing library and Rust proxy from NVIDIA that decides, per request, which LLM should handle each step of an agent workflow. It is aimed at platform and AI infrastructure teams whose agents send every call to a frontier model and are paying frontier prices for work a 30B open-weight model could finish.
NVIDIA released NeMo Switchyard on 11 August 2026 alongside the Nemotron 3.5 Lightning model. It is an Apache 2.0 licensed routing layer, written in Rust, that sits between an agent and the models it calls and picks a target per request rather than pinning the whole workflow to one model. It ships in three shapes: a standalone proxy server (cargo install switchyard-server), an embedded library (switchyard-libsy) for applications that want routing inside the process, and a CLI launcher that puts routing behind tools such as Claude Code and Codex. The reference server speaks OpenAI Chat Completions, OpenAI Responses and Anthropic Messages, and talks to vLLM, NVIDIA NIM, Ollama and any OpenAI-compatible endpoint, translating between formats so the caller does not change. Five routing algorithms ship today: an LLM Classifier that uses a judge model and keeps session affinity across turns, a Stage Router that reads recent tool activity to infer where the agent is in its lifecycle, an Escalation Router that starts cheap and promotes on sustained difficulty, plus Random and Passthrough for traffic splits and single targets. A tunable Prefill Router predicts per-model success from residual-stream signals during inference. A semantic naming layer decouples model names from provider endpoints, and every decision is logged with the chosen model, the rationale, token usage and latency. In a LangChain evaluation over 145 multi-turn tasks, routing between Nemotron 3.5 Lightning and Claude Opus 4.8 sent only 7% of calls to the frontier model and cut cost 74% (from $11.45 to $3.00 per run) for a six-point accuracy drop. Kong, LiteLLM, LangChain, Cognition, Boomi, Siemens and Cadence are named integration partners.
The platform or AI infrastructure team that already runs agents in production, has a frontier-model bill it cannot explain line by line, and has the Rust/ops skills to own a proxy in the request path.
Cut agent inference spend by routing the majority of calls to a cheap open-weight model while reserving the frontier model for the small fraction of turns that actually need it — 74% cheaper at a six-point accuracy cost in NVIDIA's published LangChain benchmark.
At a Glance
- Category
- Infrastructure & Cloud
- Pricing
- Free
- Target Market
- CTOs, Enterprise Developers, Data Scientists, Platform Engineers
- Deployment
- Open-source, Self-hosted, Hybrid
- Founded
- 1993
- Headquarters
- Santa Clara, United States
- Team Size
- 500+
Key Features
- ✓Escalation Router
Starts every task on a low-cost model and promotes to a frontier model only when an LLM judge detects sustained difficulty, which is the configuration that produced the 74% cost reduction in the LangChain benchmark.
- ✓Stage Router
Reads recent tool-call activity to infer which stage of its lifecycle an agent is in, then matches that stage to a model capability tier rather than judging each prompt in isolation.
- ✓Prefill Router
A tunable router that extracts residual-stream signals during inference to predict how likely each candidate model is to complete the task, then blends predicted accuracy against cost and latency policy.
- ✓OpenAI and Anthropic API translation
The reference server accepts OpenAI Chat Completions, OpenAI Responses and Anthropic Messages requests and translates between them, so callers keep their existing SDK while the target model changes underneath.
- ✓Semantic naming layer
Maps logical model names to provider endpoints independently, letting you swap the model behind a name without touching routing logic or application code.
- ✓Decision logging
Records the selected model, the reason it was selected, token usage, latency and outcome for every request, which is what makes routing auditable rather than a black box.
- ✓Proxy or embedded library deployment
Runs either as a standalone Rust server your agent points at or as middleware compiled into the agent process, so it fits both gateway and in-process architectures.
Capabilities
Use Cases
- •Cost-optimising a production coding agent
Route routine file edits and tool calls to an open-weight model and escalate only the hard reasoning turns, cutting the frontier-model share of calls to single digits.
- •Hybrid on-prem plus frontier deployment
Serve a fine-tuned proprietary model from your own GPUs for sensitive work and burst to a commercial frontier API for everything else, under one routing policy.
- •Model bake-offs on live traffic
Use the Random router to split real production requests across candidate models and compare accuracy, latency and cost before committing to a provider contract.
- •Governed routing behind an API gateway
Run Switchyard as the model-selection decision service behind Kong AI Gateway, which keeps credentials, policy and compliance in the gateway while Switchyard picks the target.
- •Provider failover and abstraction
Use the semantic naming layer so an outage or price change at one provider becomes a config edit rather than an application change.
Ideal For
Best For
- ✓Cutting LLM spend on multi-step agent workflows where most turns are routine and a minority are genuinely hard
- ✓Running a hybrid stack that mixes self-hosted fine-tuned open-weight models with commercial frontier APIs
- ✓Benchmarking models against each other on real production traffic before committing to one provider
- ✓Adding a provider-abstraction layer so an application can switch between OpenAI, Anthropic and NIM endpoints without a rewrite
- ✓Keeping model routing inside your own infrastructure instead of a hosted third-party router
Not Ideal For
- ✗Teams that need a production-supported product today — the GitHub repository explicitly marks Switchyard pre-alpha and experimental, and says it is not recommended for production use
- ✗Small teams without infrastructure expertise: analysts at Moor Insights & Strategy note it is not plug-and-play, existing agents will likely need re-engineering, and a misconfigured router becomes overhead rather than optimisation
- ✗Workloads where every request genuinely needs frontier reasoning — routing adds a decision hop and a judge-model call for no saving
- ✗Organisations that want a vendor SLA, commercial support and a procurement contract behind the routing layer
Integrations
Deployment
Market Analysis
Pros
- ✓Published, third-party-run benchmark evidence: LangChain measured 74% cost reduction with 7% of calls hitting the frontier model across 145 multi-turn tasks
- ✓Apache 2.0 with no paid tier, so there is no licence cost or vendor lock-in on the routing layer itself
- ✓Real integration partners shipping against it on day one, including Kong, LiteLLM, LangChain, Cognition, Boomi, Siemens and Cadence
- ✓Native OpenAI and Anthropic API translation means existing agents can point at it without an SDK change
Cons
- ✗The GitHub repository marks the project pre-alpha and experimental and states it is not recommended for production use — the marketing materials do not mention this
- ✗Moor Insights & Strategy calls it not plug-and-play: existing agents will likely need re-engineering, and getting the configuration wrong makes the router overhead rather than optimisation
- ✗LangChain flags that its own benchmark suite is saturated, with only 8 points separating the efficient and frontier models, giving routing less room to prove value than a harder workload would
- ✗Cost is not stable run to run: in the same benchmark, frontier-model usage varied between 4.1% and 9.1% across identical task sets, a 67% swing in the bill
- ✗The 74% saving came with a 6 percentage-point accuracy drop (86.0% to 80.0%), which is a real quality trade, not a free lunch
Pricing
Open source (Apache 2.0)
$0
- ✓Full routing library and Rust proxy server
- ✓All five shipped routing algorithms plus the tunable Prefill Router
- ✓OpenAI, OpenAI Responses and Anthropic API compatibility
- ✓No paid tier, no commercial support, no SLA
Switchyard itself is free under Apache 2.0 with no paid edition, no hosted plan and no commercial support contract — the real cost is the model spend it routes plus the compute for the judge model each router calls. NVIDIA publishes no list price because there is nothing to buy; the commercial pull is toward NVIDIA NIM and Nemotron as routing targets.
Security & Compliance
Connect
Sources
This page was written from 7 sources, 6 on domains other than github.com.
- 1.github.com — Switchyardvendor
- 2.developer.nvidia.com — route ai agent workloads across models with nvidia nemo swit
- 3.siliconangle.com — nvidia releases nemotron 3 5 lightning nemo switchyard give
- 4.moorinsightsstrategy.com — nvidia nemo switchyard open sourcing model routing is the re
- 5.langchain.com — switchyard agent routing benchmark
- 6.konghq.com — llm routing kong ai gateway nvidia nemo switchyard
- 7.marktechpost.com — nvidia ai releases nemotron 3 5 lightning and nemo switchyar
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
Kong AI Gateway
One gateway for LLM, MCP and agent-to-agent traffic, built on the Kong data plane you already run
WEKA NeuralMesh
Microservices storage and memory fabric built for AI training and inference at exabyte scale
Tsuga
Bring-your-own-cloud observability that keeps telemetry, and its cost, inside your own AWS account
OpenObserve
Open-source, Rust-based observability on object storage — logs, metrics, traces and LLM telemetry in one binary