LMCache Documentation — KV Cache Management for LLM Inference
by LMCache Team
Stop paying to recompute the same prefix: the KV cache layer that sits under vLLM.
Overview
LMCache is an Apache-2.0 KV cache management layer for LLM inference (11k-plus GitHub stars, 1.8k forks) that treats the KV cache as reusable, persistent state rather than throwaway GPU memory. The documentation is organised as Getting Started (overview, installation, quickstart, configuration, benchmarking), KV Cache Operations (CLI reference, HTTP API, frontend dashboard, KV Cache SDK), Recipes with model-specific configurations for uniform-attention, hybrid-attention and multimodal models, Secondary KV Storage covering backends and compression strategies, Distributed KV Cache covering prefill disaggregation, peer-to-peer sharing and coordination, Production Deployment for Kubernetes, operators and observability, and a Developer Guide. The quickstart set works through KV cache offloading, cross-instance cache sharing, disaggregated prefill, a standalone LMCacheEngine that runs without vLLM or a GPU, and multimodal caching. Documented storage backends include CPU RAM, local SSD and raw block devices, Redis and Valkey, Mooncake Store, InfiniStore, S3-compatible object storage, AWS SageMaker HyperPod, NIXL and GDS, with pluggable compression and token-dropping transformations. The distinguishing capability is non-prefix reuse: CacheBlend selectively recomputes only the portions of a reused cache that need it, which matters for RAG, where retrieved chunks land in the middle of a prompt rather than at its start. Reported gains are workload-dependent and should be read that way — the team's arXiv paper (2510.09665, October 2025) claims up to 15x throughput improvement on multi-round question answering and document analysis when paired with vLLM, while a 2026 AMD MI300X agentic benchmark reports 3.0x lower average time-to-first-token at 32 concurrent users and 100K context. Development is supported in part by Tensormesh.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-5 hours from quickstart to a Kubernetes deployment; reference docs are ongoing
- Provider
- LMCache Team
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Offload KV cache from GPU memory to CPU RAM and local disk
- ✓Configure Redis, Mooncake, NIXL or S3 as a secondary cache backend
- ✓Reuse non-prefix KV segments for RAG workloads using CacheBlend
- ✓Set up disaggregated prefill so prefill and decode scale independently
- ✓Share cached prefixes peer-to-peer across multiple serving instances
- ✓Deploy LMCache on Kubernetes with metrics and cache-hit observability
- ✓Benchmark TTFT and throughput before and after enabling the cache
Highlights
- •Non-prefix reuse via CacheBlend goes beyond the prefix caching engines already do
- •The cache survives an engine crash because storage is decoupled from the engine
- •Documents a standalone engine you can try without vLLM or a GPU at all
- •Nine-plus storage backends documented, including S3, Mooncake, InfiniStore, NIXL and GDS
- •Backed by a public arXiv paper rather than marketing benchmarks alone
Who It’s For
Best For
- ✓Inference platform engineers serving long-context or multi-turn workloads
- ✓Teams running RAG where retrieved context repeats across many requests
- ✓Anyone operating vLLM at scale and paying for redundant prefill compute
- ✓Agent infrastructure teams whose prompts grow across dozens of turns
Prerequisites
- •Hands-on experience serving models with vLLM or a comparable engine
- •Understanding of KV cache, prefill versus decode, and time-to-first-token
- •Kubernetes familiarity for the production deployment sections
FAQ
What is LMCache Documentation — KV Cache Management for LLM Inference?
Official documentation for LMCache, the Apache-2.0 KV cache management layer that decouples cache storage from the inference engine so prefixes survive across requests, across engine instances, and even across an engine crash. It covers tiered offloading from GPU to CPU RAM, disk, Redis, Mooncake, NIXL and S3, non-prefix reuse via CacheBlend, peer-to-peer sharing, disaggregated prefill, and Kubernetes deployment with observability.
Is LMCache Documentation — KV Cache Management for LLM Inference free?
LMCache Documentation — KV Cache Management for LLM Inference is free to access.
What level is LMCache Documentation — KV Cache Management for LLM Inference for?
LMCache Documentation — KV Cache Management for LLM Inference is aimed at a advanced audience. Recommended background: Hands-on experience serving models with vLLM or a comparable engine, Understanding of KV cache, prefill versus decode, and time-to-first-token, Kubernetes familiarity for the production deployment sections.
How long does LMCache Documentation — KV Cache Management for LLM Inference take?
Expect roughly ~3-5 hours from quickstart to a Kubernetes deployment; reference docs are ongoing. Most learners work through it at their own pace.
What will I learn from LMCache Documentation — KV Cache Management for LLM Inference?
You'll learn: Offload KV cache from GPU memory to CPU RAM and local disk; Configure Redis, Mooncake, NIXL or S3 as a secondary cache backend; Reuse non-prefix KV segments for RAG workloads using CacheBlend; Set up disaggregated prefill so prefill and decode scale independently; Share cached prefixes peer-to-peer across multiple serving instances; Deploy LMCache on Kubernetes with metrics and cache-hit observability; Benchmark TTFT and throughput before and after enabling the cache.
Topics
Sources
This page was written from 4 sources, 2 on domains other than docs.lmcache.ai.