llm-d Documentation — Kubernetes-Native Distributed LLM Inference

by llm-d (CNCF Sandbox)

AdvancedDocumentationFree~3-5 hours to read the well-lit paths end to end; budget a day to run the quickstart against a real cluster

Tested deployment recipes for turning single-node vLLM into a multi-node serving stack on Kubernetes.

Start LearningAdded Aug 9, 2026 · Updated Aug 9, 2026

Overview

llm-d is an open-source, Kubernetes-native distributed inference stack that runs vLLM, SGLang and similar engines across a cluster, and its documentation is structured around what it calls well-lit paths — tested deployment recipes rather than exhaustive reference. The docs split into Getting Started (quickstart plus accelerator support), Well-Lit Paths, Concepts and Architecture, Operations & Monitoring, and Infrastructure & Environments. The Foundations paths walk through a baseline deployment, latency-based request routing, prefix-cache-aware routing, tiered caching, prefill/decode disaggregation, mixture-of-experts scaling with expert parallelism, flow control and fairness, and autoscaling; the Workloads paths cover agentic applications, multimodal models and batch processing. Architecturally the stack centres on three pieces: a Router built from a high-performance L7 proxy (typically Envoy) plus an Endpoint Picker that scores servers on live metrics and policy, an InferencePool that groups model servers by base model, and the model servers themselves. Optional layers add KV-cache indexing and offload to CPU or SSD, latency prediction for routing, OpenAI-compatible batch APIs, and workload-aware autoscaling. It is Apache-2.0 licensed, sits at roughly 4k GitHub stars, and was founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA with support from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. It was accepted as a CNCF Sandbox project in March 2026; the current v0.8 release graduates multimodal, batch and flow-control to production, following v0.7 in May 2026 and v0.5's hierarchical KV offloading in February 2026. Published figures include roughly 3x higher output throughput and 2x faster time-to-first-token versus a baseline, across GPUs, TPUs, XPUs, CPUs and NPUs.

At a Glance

Topic: Frameworks
Level: Advanced
Format: Documentation
Cost: Free
Duration: ~3-5 hours to read the well-lit paths end to end; budget a day to run the quickstart against a real cluster
Provider: llm-d (CNCF Sandbox)
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓Deploy a baseline distributed inference service on Kubernetes and benchmark it properly
✓Route requests by prefix-cache locality and measured latency rather than round-robin
✓Offload KV cache in tiers to CPU and SSD to raise effective context capacity
✓Disaggregate prefill and decode onto separate pools and understand when that actually helps
✓Scale mixture-of-experts models using expert parallelism across nodes
✓Apply flow control and fairness so one heavy tenant cannot starve the rest
✓Configure SLO-aware, workload-aware autoscaling instead of naive CPU-based HPA
✓Serve agentic, multimodal and batch workloads, including OpenAI-compatible batch APIs

Highlights

•Well-lit paths are tested recipes with benchmarks, not aspirational architecture diagrams
•Covers the operational half of inference that vLLM's own docs deliberately leave out — routing, fairness, autoscaling, monitoring
•Vendor-neutral: founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, and a CNCF Sandbox project since March 2026
•Portable across GPUs, TPUs, XPUs, CPUs and NPUs, so the patterns survive an accelerator change
•Release notes are specific and dated (v0.8 current, v0.7 in May 2026, hierarchical KV offload in v0.5), which makes it easy to tell what is actually production-ready

Who It’s For

Best For

✓Platform and MLOps engineers running self-hosted LLM inference on Kubernetes
✓Inference engineers chasing throughput and TTFT beyond what single-node vLLM gives
✓Architects evaluating disaggregated serving and KV-cache offloading for long-context workloads
✓Teams serving agentic or batch traffic where request shapes vary wildly

Prerequisites

•Solid Kubernetes operational experience — Deployments, Services, Gateway API, kustomize
•Hands-on familiarity with vLLM or SGLang on a single node
•Understanding of LLM inference mechanics: KV cache, prefill vs decode, batching, TTFT

FAQ

What is llm-d Documentation — Kubernetes-Native Distributed LLM Inference?

llm-d's documentation is the reference for running LLM inference at production scale on Kubernetes, aimed at platform and inference engineers who already have vLLM or SGLang working on one node and now need prefix-cache-aware routing, KV-cache offloading, prefill/decode disaggregation and autoscaling across a cluster. Rather than an API dump it is organised as tested 'well-lit paths' you can adapt, so you finish able to deploy and benchmark a distributed serving stack instead of only understanding one in theory.

Is llm-d Documentation — Kubernetes-Native Distributed LLM Inference free?

llm-d Documentation — Kubernetes-Native Distributed LLM Inference is free to access.

What level is llm-d Documentation — Kubernetes-Native Distributed LLM Inference for?

llm-d Documentation — Kubernetes-Native Distributed LLM Inference is aimed at a advanced audience. Recommended background: Solid Kubernetes operational experience — Deployments, Services, Gateway API, kustomize, Hands-on familiarity with vLLM or SGLang on a single node, Understanding of LLM inference mechanics: KV cache, prefill vs decode, batching, TTFT.

How long does llm-d Documentation — Kubernetes-Native Distributed LLM Inference take?

Expect roughly ~3-5 hours to read the well-lit paths end to end; budget a day to run the quickstart against a real cluster. Most learners work through it at their own pace.

What will I learn from llm-d Documentation — Kubernetes-Native Distributed LLM Inference?

You'll learn: Deploy a baseline distributed inference service on Kubernetes and benchmark it properly; Route requests by prefix-cache locality and measured latency rather than round-robin; Offload KV cache in tiers to CPU and SSD to raise effective context capacity; Disaggregate prefill and decode onto separate pools and understand when that actually helps; Scale mixture-of-experts models using expert parallelism across nodes; Apply flow control and fairness so one heavy tenant cannot starve the rest; Configure SLO-aware, workload-aware autoscaling instead of naive CPU-based HPA; Serve agentic, multimodal and batch workloads, including OpenAI-compatible batch APIs.

Topics

llm-inferencekubernetesvllmkv-cachedisaggregated-servingcncf

Sources

This page was written from 3 sources, 1 on domains other than llm-d.ai.