KServe Documentation — Generative and Predictive AI Inference on Kubernetes
by KServe (CNCF)
The CNCF-standard way to run vLLM and llm-d on Kubernetes — declarative CRDs, KV-cache-aware routing and inference-aware autoscaling, documented end to end.
Overview
The KServe docs are organised as Getting Started, Concepts, Model Serving, Install & Operations, Reference and Community, with Model Serving split into Generative AI, Predictive AI, Inference Graphs and storage. The current documented release is 0.20, and the generative half reflects KServe's GenAI-first shift: LLMInferenceService, a CRD built on the llm-d framework with vLLM as the inference engine. That CRD is documented across four specification areas — model (Hugging Face, S3 or PVC sources, plus optional LoRA adapters for multi-tenant serving), workload (single-node templates, multi-node workers, prefill-only pools), router (Gateway API entry points, path rules, endpoint selection) and parallelism (tensor, data and expert parallelism for MoE models). Supporting pages cover LoRA adapters, KV cache offloading, model cache, config composition, dependencies, status reference, traffic management, autoscaling and rate limiting through an AI gateway. Four deployment patterns are described in detail: single-node for models under roughly 7B, multi-node via LeaderWorkerSet, prefill/decode disaggregation for cost, and the parallelism variants. Routing is KV-cache-aware and load-aware, and scaling uses the Workload Variant Autoscaler rather than CPU metrics. The predictive side still serves TensorFlow, PyTorch, scikit-learn, XGBoost and ONNX from the same control plane. KServe began in 2019 as a Google, IBM, Bloomberg, NVIDIA and Seldon collaboration under Kubeflow, moved to LF AI & Data in February 2022, and was accepted as a CNCF incubating project in September 2025 with 19 maintainers, 300+ contributors and 30+ named adopters including Bloomberg, Red Hat, SAP and NVIDIA.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~4-6 hours for the quickstarts plus the generative-inference guides; open-ended as reference
- Provider
- KServe (CNCF)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Deploy an OpenAI-compatible LLM endpoint by applying a single InferenceService custom resource
- ✓Choose between single-node, multi-node LeaderWorkerSet and prefill/decode disaggregated serving topologies
- ✓Configure tensor, data and expert parallelism for MoE models across GPUs and nodes
- ✓Serve many LoRA adapters from one base model for multi-tenant inference workloads
- ✓Route traffic using KV-cache-aware, prefix-cache and load-aware scheduling on the Gateway API
- ✓Offload KV cache and warm a model cache to cut cold starts and memory pressure
- ✓Scale on inference-specific signals with the Workload Variant Autoscaler instead of CPU metrics
- ✓Serve classical TensorFlow, PyTorch, scikit-learn, XGBoost and ONNX models from the same platform
Highlights
- •The only Kubernetes model-serving CRD accepted as a CNCF incubating project, with neutral governance since September 2025
- •Generative-inference docs build on llm-d and vLLM rather than a bespoke runtime, so the tuning knobs you learn transfer
- •The GenAI quickstart deploys a real model — Qwen2.5-0.5B-Instruct on one GPU — and verifies a chat-completion response, not a toy echo server
- •Covers the operational surface most serving tutorials skip: traffic management, status reference, dependency wiring and gateway rate limiting
- •Predictive and generative serving share one control plane, which matters if you still run classical models alongside LLMs
Who It’s For
Best For
- ✓Platform engineers who must run LLM inference on an existing Kubernetes cluster
- ✓ML infrastructure teams standardising on vLLM or llm-d across many models
- ✓SREs replacing hand-written Deployments and Ingress rules with a declarative serving CRD
- ✓Organisations serving both classical ML models and LLMs from a single platform
Prerequisites
- •Working Kubernetes knowledge: CRDs, Deployments, Services and the Gateway API
- •Familiarity with vLLM or another OpenAI-compatible inference server
- •Access to a GPU-enabled cluster or node pool to follow the quickstarts
FAQ
What is KServe Documentation — Generative and Predictive AI Inference on Kubernetes?
KServe is the CNCF-incubating model-serving platform for Kubernetes, and this is its official documentation. It is written for platform and ML infrastructure engineers who already run Kubernetes and now have to serve LLMs in production with autoscaling, canary rollouts, LoRA multi-tenancy and cache-aware routing. Work through it and you can stand up an OpenAI-compatible inference endpoint from a single declarative custom resource instead of a hand-rolled Deployment, Service and Ingress.
Is KServe Documentation — Generative and Predictive AI Inference on Kubernetes free?
KServe Documentation — Generative and Predictive AI Inference on Kubernetes is free to access.
What level is KServe Documentation — Generative and Predictive AI Inference on Kubernetes for?
KServe Documentation — Generative and Predictive AI Inference on Kubernetes is aimed at a advanced audience. Recommended background: Working Kubernetes knowledge: CRDs, Deployments, Services and the Gateway API, Familiarity with vLLM or another OpenAI-compatible inference server, Access to a GPU-enabled cluster or node pool to follow the quickstarts.
How long does KServe Documentation — Generative and Predictive AI Inference on Kubernetes take?
Expect roughly ~4-6 hours for the quickstarts plus the generative-inference guides; open-ended as reference. Most learners work through it at their own pace.
What will I learn from KServe Documentation — Generative and Predictive AI Inference on Kubernetes?
You'll learn: Deploy an OpenAI-compatible LLM endpoint by applying a single InferenceService custom resource; Choose between single-node, multi-node LeaderWorkerSet and prefill/decode disaggregated serving topologies; Configure tensor, data and expert parallelism for MoE models across GPUs and nodes; Serve many LoRA adapters from one base model for multi-tenant inference workloads; Route traffic using KV-cache-aware, prefix-cache and load-aware scheduling on the Gateway API; Offload KV cache and warm a model cache to cut cold starts and memory pressure; Scale on inference-specific signals with the Workload Variant Autoscaler instead of CPU metrics; Serve classical TensorFlow, PyTorch, scikit-learn, XGBoost and ONNX models from the same platform.
Topics
Sources
This page was written from 4 sources, 2 on domains other than kserve.github.io.