LMCache Documentation — KV Cache Management for LLM Inference
by LMCache (PyTorch Foundation)
Stop recomputing the same prefill — persist, tier and reuse KV cache across requests, sessions and serving engines.
Overview
LMCache is an Apache-2.0 licensed KV cache management layer for LLM inference that runs as a standalone daemon independent of the inference engine, and it joined the PyTorch Foundation in October 2025. Its core idea is to treat KV cache as reusable, persistent knowledge rather than throwaway per-request state: caches can be offloaded from GPU memory into tiered storage (CPU RAM, local disk, Redis, S3, NIXL and other backends), reused across requests and sessions, shared between multiple serving instances, and survive an inference-engine crash. Beyond standard prefix caching it supports non-prefix KV reuse via CacheBlend, prefill-decode disaggregation over NVLink, RDMA or TCP, pluggable compression and serialization, and production-grade observability with Kubernetes and KV-cache-specific metrics. It runs either in-process inside vLLM through LMCacheConnectorV1 — configured by environment variables or a YAML file, the simplest path to single-node CPU/disk offloading — or in multiprocess mode where a standalone LMCache server owns the storage and multiple vLLM instances attach via LMCacheMPConnector, which is the recommended setup for distributed KV storage and cross-instance sharing. vLLM ships a built-in shortcut (`--kv-offloading-backend lmcache` with `--kv-offloading-size`) that wires this up for you. The documented payoff is lower time-to-first-token and higher throughput on long-context and multi-turn workloads; the multiprocess architecture released in April 2026 reported roughly 10x gains for MoE inference, and May 2026 added agentic-workload benchmarking on AMD MI300X.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~2-3 hours to read the guides and run the quickstart
- Provider
- LMCache (PyTorch Foundation)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Offload KV cache from GPU to CPU, local disk or remote backends such as Redis and S3 to raise effective cache capacity
- ✓Choose between in-process (LMCacheConnectorV1) and multiprocess (LMCacheMPConnector) deployment and configure each
- ✓Reuse KV cache across requests, sessions and multiple vLLM instances to cut repeated prefill cost
- ✓Apply non-prefix KV reuse with CacheBlend for RAG-style prompts where the shared text is not a clean prefix
- ✓Set up prefill-decode disaggregation over NVLink, RDMA or TCP
- ✓Monitor cache hit behaviour in production with the Kubernetes and KV-cache-specific metrics
Highlights
- •Engine-independent daemon — KV cache survives an inference-engine crash and is shareable across instances
- •Non-prefix reuse (CacheBlend) goes beyond the prefix caching built into most serving engines
- •Part of the PyTorch Foundation since October 2025, Apache 2.0, vendor-neutral across storage and hardware
- •First-class vLLM integration, including a one-flag setup path
Who It’s For
Best For
- ✓Inference engineers optimizing TTFT and throughput for long-context serving
- ✓Teams running multi-turn chat or agentic workloads with heavy prompt reuse
- ✓Platform engineers operating vLLM at scale across multiple replicas
- ✓Anyone trying to cut GPU cost on prefill-dominated traffic
Prerequisites
- •Experience serving LLMs in production, ideally with vLLM
- •Understanding of transformer inference internals — prefill vs decode, KV cache, attention
- •Comfort with Kubernetes or containerized deployment for the production setups
FAQ
What is LMCache Documentation — KV Cache Management for LLM Inference?
Official documentation for LMCache, the open-source KV cache management layer that offloads and reuses LLM key-value caches across GPU, CPU, disk and remote storage. It is aimed at engineers serving long-context, multi-turn or agentic workloads where repeated prefill dominates latency and cost.
Is LMCache Documentation — KV Cache Management for LLM Inference free?
LMCache Documentation — KV Cache Management for LLM Inference is free to access.
What level is LMCache Documentation — KV Cache Management for LLM Inference for?
LMCache Documentation — KV Cache Management for LLM Inference is aimed at a advanced audience. Recommended background: Experience serving LLMs in production, ideally with vLLM, Understanding of transformer inference internals — prefill vs decode, KV cache, attention, Comfort with Kubernetes or containerized deployment for the production setups.
How long does LMCache Documentation — KV Cache Management for LLM Inference take?
Expect roughly ~2-3 hours to read the guides and run the quickstart. Most learners work through it at their own pace.
What will I learn from LMCache Documentation — KV Cache Management for LLM Inference?
You'll learn: Offload KV cache from GPU to CPU, local disk or remote backends such as Redis and S3 to raise effective cache capacity; Choose between in-process (LMCacheConnectorV1) and multiprocess (LMCacheMPConnector) deployment and configure each; Reuse KV cache across requests, sessions and multiple vLLM instances to cut repeated prefill cost; Apply non-prefix KV reuse with CacheBlend for RAG-style prompts where the shared text is not a clean prefix; Set up prefill-decode disaggregation over NVLink, RDMA or TCP; Monitor cache hit behaviour in production with the Kubernetes and KV-cache-specific metrics.