NVIDIA Dynamo Documentation
by NVIDIA
Serve LLMs across many nodes with disaggregated prefill/decode and KV-cache-aware routing.
Overview
Dynamo is NVIDIA's Kubernetes-native, inference-engine-agnostic serving layer: it sits above SGLang, TensorRT-LLM, and vLLM rather than replacing them, and is fully open source (built in Rust for the data path and Python for extensibility). Its central ideas are disaggregated serving — splitting prefill and decode onto separate worker pools so each can be scaled and scheduled independently — plus a KV-cache-aware router that steers requests to the worker most likely to have a usable prefix, multi-tier KV cache offloading, dynamic GPU scheduling, and accelerated inter-node data transfer. The documentation covers a container fast-path quickstart, local installation, Kubernetes deployment, building from source, architecture and design docs, per-component guides for the Frontend and Router, an OpenAI-compatible multi-model routing API, worker discovery, autoscaling and scheduling, observability and health checks, and optimizations specifically for agentic inference workloads. Container images referenced in the docs are at version 1.3.0.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced; quickstart in ~30 min, full deployment guides longer
- Provider
- NVIDIA
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Deploy disaggregated prefill/decode LLM serving across multiple nodes
- ✓Configure KV-cache-aware routing so prefix reuse actually lowers time-to-first-token
- ✓Run Dynamo on top of vLLM, SGLang, or TensorRT-LLM without rewriting your serving stack
- ✓Set up Kubernetes deployments with autoscaling, worker discovery, and health checks
- ✓Expose an OpenAI-compatible endpoint that routes across multiple models
- ✓Instrument inference with observability suited to agentic, multi-turn workloads
Highlights
- •Engine-agnostic: keeps your existing vLLM/SGLang/TensorRT-LLM investment
- •Disaggregated prefill and decode is the current state of the art for high-throughput serving
- •Fully open source with production Kubernetes deployment recipes
- •Explicit treatment of agentic inference patterns, not just single-shot completions
Who It’s For
Best For
- ✓Inference and platform engineers running LLMs at datacenter scale
- ✓Teams hitting throughput or TTFT limits with single-process vLLM/SGLang
- ✓MLOps engineers designing Kubernetes-native GPU serving infrastructure
- ✓Anyone optimizing cost per token for reasoning or agentic workloads
Prerequisites
- •Experience serving LLMs with vLLM, SGLang, or TensorRT-LLM
- •Working Kubernetes and container knowledge
- •Understanding of KV cache, prefill vs. decode, and GPU memory constraints
FAQ
What is NVIDIA Dynamo Documentation?
Official documentation for NVIDIA Dynamo, an open-source, backend-agnostic inference framework for serving LLMs and reasoning models across multi-node GPU clusters. Aimed at inference and platform engineers who have outgrown a single vLLM or SGLang process.
Is NVIDIA Dynamo Documentation free?
NVIDIA Dynamo Documentation is free to access.
What level is NVIDIA Dynamo Documentation for?
NVIDIA Dynamo Documentation is aimed at a advanced audience. Recommended background: Experience serving LLMs with vLLM, SGLang, or TensorRT-LLM, Working Kubernetes and container knowledge, Understanding of KV cache, prefill vs. decode, and GPU memory constraints.
How long does NVIDIA Dynamo Documentation take?
Expect roughly Self-paced; quickstart in ~30 min, full deployment guides longer. Most learners work through it at their own pace.
What will I learn from NVIDIA Dynamo Documentation?
You'll learn: Deploy disaggregated prefill/decode LLM serving across multiple nodes; Configure KV-cache-aware routing so prefix reuse actually lowers time-to-first-token; Run Dynamo on top of vLLM, SGLang, or TensorRT-LLM without rewriting your serving stack; Set up Kubernetes deployments with autoscaling, worker discovery, and health checks; Expose an OpenAI-compatible endpoint that routes across multiple models; Instrument inference with observability suited to agentic, multi-turn workloads.