Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving
by Mooncake Team (Moonshot AI / kvcache-ai)
The docs for the KV cache transfer engine and distributed store behind Kimi, with integration guides for vLLM, SGLang, LMCache and LMDeploy.
Overview
Mooncake splits prefill and decode into separate clusters and builds a disaggregated KV cache pool out of the CPU, DRAM and SSD capacity that GPU clusters usually leave idle. A KVCache-centric scheduler balances throughput against latency SLOs and uses prediction-based early rejection under overload; the project reports up to a 525% throughput increase in simulated long-context scenarios and 75% more requests handled for Kimi on real workloads, and the design won a FAST 2025 Best Paper award. The documentation is organized into Getting Started (quick start, build guide, Store-RS quickstart, features), Deployment (Store deployment and tuning, Kubernetes, and integration guides for SGLang, vLLM, LMCache and LMDeploy), Performance benchmarks, Design Documents (architecture, Transfer Engine, TENT, Mooncake Store, Store-RS, PG, EP, Conductor, Reshard, P2P Store, SGLang HiCache), API references for Python, C/C++, HTTP and Rust, and Troubleshooting with an error-code guide. The quick start installs the mooncake-transfer-engine package (a separate cuda13 build exists), starts mooncake_master on port 50051, and runs a Python put/get against the store using P2PHANDSHAKE mode. The Transfer Engine supports TCP, RDMA, AWS EFA, NVMe-oF, NVLink and CXL among other transports. The Transfer Engine was open-sourced in November 2024 and Mooncake Store in March 2025; the project joined the PyTorch Ecosystem in February 2026 and the docs list integrations added through August 2026.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~30 min for the single-node quick start; a few days to work through the design documents and one framework integration (estimate)
- Provider
- Mooncake Team (Moonshot AI / kvcache-ai)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Understand how prefill/decode disaggregation and a pooled KV cache change LLM serving throughput
- ✓Install the Transfer Engine package and start a local Mooncake Store master service
- ✓Store and retrieve objects from Mooncake Store through the Python client API
- ✓Configure SGLang or vLLM for direct KV transfer between prefill and decode workers
- ✓Use Mooncake Store as a distributed KV cache backend for SGLang HiCache or vLLM's MooncakeStoreConnector
- ✓Deploy Mooncake on Kubernetes and tune it for high availability and SSD offload
- ✓Choose between TCP, RDMA, EFA and NVLink transports based on your cluster's network hardware
- ✓Diagnose failures with the error-code reference and key access path debugging guide
Highlights
- •Production system: it is the serving platform for Kimi, and the design paper won Best Paper at FAST 2025
- •Integrated with SGLang, vLLM, TensorRT-LLM, LMCache, LMDeploy and NIXL, and used by RL and training projects such as checkpoint-engine and vLLM's Speculators for multi-node hidden-state streaming
- •The repo has 6.8k stars and 1.3k forks under Apache-2.0, with news items running to August 2026
- •Published Transfer Engine numbers: 87 GB/s on 4x200 Gbps RoCE and 190 GB/s on 8x400 Gbps RoCE, about 2.4x and 4.6x faster than TCP
- •Covers unusually broad hardware, including CUDA, ROCm, Ascend NPUs, AWS EFA and several other accelerator families
Who It’s For
Best For
- ✓Inference platform engineers serving long-context LLMs on multi-node GPU clusters
- ✓Teams running vLLM or SGLang who want prefill/decode disaggregation or KV cache reuse across requests
- ✓Infrastructure engineers evaluating RDMA and high-speed transfer layers for LLM serving
- ✓Systems researchers studying KV cache scheduling and storage tiering
Prerequisites
- •Experience running an LLM serving engine such as vLLM or SGLang
- •Familiarity with KV caching, prefill versus decode, and cluster networking (RDMA is optional but central to the performance story)
- •Python 3.10+ and, for GPU transfers, CUDA 12.1+ with GPUDirect
FAQ
What is Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving?
The official documentation for Mooncake, the open-source KVCache-centric serving architecture that powers Moonshot AI's Kimi. It is for inference and platform engineers running LLMs on GPU clusters who want prefill/decode disaggregation or a shared KV cache pool. Afterwards you can deploy Mooncake Store, connect it to vLLM or SGLang, and tune transfers.
Is Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving free?
Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving is free to access.
What level is Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving for?
Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving is aimed at a advanced audience. Recommended background: Experience running an LLM serving engine such as vLLM or SGLang, Familiarity with KV caching, prefill versus decode, and cluster networking (RDMA is optional but central to the performance story), Python 3.10+ and, for GPU transfers, CUDA 12.1+ with GPUDirect.
How long does Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving take?
Expect roughly ~30 min for the single-node quick start; a few days to work through the design documents and one framework integration (estimate). Most learners work through it at their own pace.
What will I learn from Mooncake Documentation — KVCache-Centric Disaggregated LLM Serving?
You'll learn: Understand how prefill/decode disaggregation and a pooled KV cache change LLM serving throughput; Install the Transfer Engine package and start a local Mooncake Store master service; Store and retrieve objects from Mooncake Store through the Python client API; Configure SGLang or vLLM for direct KV transfer between prefill and decode workers; Use Mooncake Store as a distributed KV cache backend for SGLang HiCache or vLLM's MooncakeStoreConnector; Deploy Mooncake on Kubernetes and tune it for high availability and SSD offload; Choose between TCP, RDMA, EFA and NVLink transports based on your cluster's network hardware; Diagnose failures with the error-code reference and key access path debugging guide.
Topics
Sources
This page was written from 4 sources, 2 on domains other than kvcache-ai.github.io.