NVIDIA Dynamo Documentation

NVIDIA Dynamo is an Apache-2.0 datacenter-scale inference serving framework that sits above an existing engine — vLLM, SGLang or TensorRT-LLM — rather than replacing it, and this documentation set is its reference. The architecture docs split the system into three planes. The request plane carries the hot path: an OpenAI-compatible frontend, a router that picks workers by both load and KV-cache overlap, prefill workers that compute prompt KV state, and decode workers that generate tokens. The control plane holds the Planner, an SLA-driven autoscaler that sizes prefill and decode pools independently from live metrics, the Dynamo Kubernetes Operator, service discovery, the Grove topology-aware scheduler for multinode placement, and ModelExpress for streaming weights between GPUs to cut cold starts. The storage and events plane carries KV Events, the KV Block Manager that offloads and recalls blocks across GPU, CPU, SSD and remote storage, and NIXL, the transfer library that moves KV cache over NVLink, InfiniBand RDMA or TCP. The docs trace the disaggregated request flow step by step, then branch into router design, planner design, fault tolerance with canary health checks and in-flight request migration, a per-backend feature matrix, CLI installation and a full Kubernetes guide. Version 1.4.0 ships prebuilt per-backend runtime containers and a `uv pip install "ai-dynamo[sglang]"` path. It is written in Rust with Python extensibility; the repository carries roughly 7.8k stars and 160-plus contributors. Independent write-ups flag the practical catch: NIXL wants a capable interconnect, and the TCP fallback costs real latency.

Topic

Frameworks

Level

Advanced

Format

Documentation

Cost

Free

Duration

Reference docs; ~3-4 hours to work through the architecture and Kubernetes deployment guides

Provider

NVIDIA

Hands-on

Yes — code/exercises

Certificate

None

• ✓How disaggregated prefill and decode let each phase scale independently
• ✓How KV-cache-aware routing avoids recomputing prompt state across workers
• ✓What the KV Block Manager offloads to CPU, SSD and remote storage
• ✓How NIXL moves KV cache over NVLink, InfiniBand RDMA or plain TCP
• ✓How the SLA-driven Planner autoscales prefill and decode pools separately
• ✓How to deploy Dynamo on Kubernetes with its operator and Grove scheduler
• ✓Which capabilities each inference backend supports via the feature matrix

• •Layers over vLLM, SGLang and TensorRT-LLM instead of replacing your engine
• •Architecture is documented as three explicit planes with a traced request flow
• •Apache 2.0, roughly 7.8k GitHub stars and more than 160 contributors
• •Covers fault tolerance: canary health checks and in-flight request migration
• •A per-backend feature matrix tells you what actually works before you commit

This page was written from 3 sources, 2 on domains other than docs.nvidia.com.

1. 1.docs.nvidia.com — overall architecturevendor
2. 2.github.com — dynamo
3. 3.spheron.network — nvidia dynamo disaggregated inference guide