LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs
by OpenMMLab / InternLM
Two inference engines under one CLI — a C++/CUDA one for throughput and a pure-Python one you can actually read and patch.
Overview
LMDeploy is a compression, deployment and serving toolkit for LLMs and vision-language models, built by the MMRazor and MMDeploy teams behind OpenMMLab and released under Apache 2.0. Its distinguishing design choice is two interchangeable inference engines: TurboMind, written in C++/CUDA for maximum throughput, and a pure-Python PyTorch engine that is far easier to read, patch and extend when you need custom behaviour. The documentation is organised as Get Started (installation, quick start, other platforms), Models (supported models, reward models), LLM Deployment (offline inference, an OpenAI-compatible server, tool calling, reasoning models, LoRA serving, request distribution), VLM Deployment (offline multimodal inference and multimodal endpoints), Quantization (AWQ/GPTQ, SmoothQuant, KV-cache quantization, llm-compressor formats), Benchmark (throughput benchmarking plus model evaluation through OpenCompass and VLMEvalKit), an Advanced Guide (TurboMind and PyTorch engine architectures, context-length extension, chat templates, debugging, structured output, multi-node deployment, profiling, metrics and speculative decoding) and an API Reference covering the inference pipeline, OpenAI-compatible endpoints and CLI tools. The project's own benchmarks claim up to 1.8x the request throughput of vLLM via persistent batching and blocked KV cache, 4-bit inference at roughly 2.4x FP16, and MXFP4 on Hopper at about 1.5x vLLM — treat those as vendor numbers and re-run the documented benchmark on your own hardware and traffic shape. It supports 45+ LLM architectures and 30+ VLMs, requires CUDA 11.3 or newer, and runs on Linux and Windows. Version 0.16.0 shipped on 19 August 2026 with wheels for Python 3.10 through 3.14.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~1-2 hours to a served OpenAI-compatible endpoint; longer for quantization and multi-node work
- Provider
- OpenMMLab / InternLM
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Stand up an OpenAI-compatible inference server for an open-weight model from a single CLI command
- ✓Choose between the TurboMind C++/CUDA engine and the pure-Python PyTorch engine, and understand the trade-off
- ✓Apply weight-only quantization with AWQ, GPTQ, SmoothQuant or llm-compressor formats and measure the accuracy cost
- ✓Quantize the KV cache and combine it with automatic prefix caching and AWQ in the same deployment
- ✓Serve vision-language models and multimodal endpoints alongside text models on the same stack
- ✓Serve multiple LoRA adapters and distribute requests across several models and machines
- ✓Enable speculative decoding, extend context length and configure chat templates for a specific model family
- ✓Benchmark throughput reproducibly and evaluate quality through OpenCompass and VLMEvalKit
Highlights
- •The dual-engine design is the reason to pick it: optimise with TurboMind, then debug or extend in readable Python without changing tools
- •Quantization is first-class rather than an afterthought — weight-only, SmoothQuant and KV-cache quantization each get their own guide
- •Ships with both a throughput benchmark and a quality-evaluation path (OpenCompass, VLMEvalKit), so speed claims are checkable
- •Text and vision-language deployment are documented symmetrically, which is rare among inference servers
- •Actively maintained by the OpenMMLab teams: v0.16.0 landed on 19 August 2026 with Python 3.10-3.14 wheels, Apache 2.0
Who It’s For
Best For
- ✓Inference and platform engineers self-hosting open-weight models under real latency or cost pressure
- ✓Teams squeezing a large model onto constrained GPUs via 4-bit weights and a quantized KV cache
- ✓Engineers serving vision-language models who want the same tooling they use for text
- ✓Anyone who needs to modify inference internals and would rather patch Python than CUDA
Prerequisites
- •An NVIDIA GPU with CUDA 11.3 or newer, plus comfort managing CUDA and Python environments
- •Working understanding of LLM inference concepts — KV cache, batching, quantization formats, tensor parallelism
- •Familiarity with an OpenAI-compatible API surface if you plan to drop this behind existing application code
FAQ
What is LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs?
LMDeploy is a compression, deployment and serving toolkit for large language and vision-language models from the OpenMMLab teams behind MMRazor and MMDeploy. It is written for engineers who have to put an open-weight model behind a real endpoint and care about tokens per second and VRAM. The docs walk from installation to an OpenAI-compatible server, then into AWQ/GPTQ and KV-cache quantization, LoRA serving, speculative decoding, multi-node deployment and reproducible benchmarking, for both text and multimodal models.
Is LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs free?
LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs is free to access.
What level is LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs for?
LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs is aimed at a advanced audience. Recommended background: An NVIDIA GPU with CUDA 11.3 or newer, plus comfort managing CUDA and Python environments, Working understanding of LLM inference concepts — KV cache, batching, quantization formats, tensor parallelism, Familiarity with an OpenAI-compatible API surface if you plan to drop this behind existing application code.
How long does LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs take?
Expect roughly ~1-2 hours to a served OpenAI-compatible endpoint; longer for quantization and multi-node work. Most learners work through it at their own pace.
What will I learn from LMDeploy Documentation — Compressing, Deploying and Serving LLMs and VLMs?
You'll learn: Stand up an OpenAI-compatible inference server for an open-weight model from a single CLI command; Choose between the TurboMind C++/CUDA engine and the pure-Python PyTorch engine, and understand the trade-off; Apply weight-only quantization with AWQ, GPTQ, SmoothQuant or llm-compressor formats and measure the accuracy cost; Quantize the KV cache and combine it with automatic prefix caching and AWQ in the same deployment; Serve vision-language models and multimodal endpoints alongside text models on the same stack; Serve multiple LoRA adapters and distribute requests across several models and machines; Enable speculative decoding, extend context length and configure chat templates for a specific model family; Benchmark throughput reproducibly and evaluate quality through OpenCompass and VLMEvalKit.
Topics
Sources
This page was written from 3 sources, 2 on domains other than lmdeploy.readthedocs.io.