vLLM Documentation
by vLLM Project
The reference manual for the inference engine most self-hosted LLM stacks are actually built on.
Overview
vLLM began at UC Berkeley's Sky Computing Lab as the reference implementation of PagedAttention — managing the KV cache like virtual memory pages so a server can pack many concurrent sequences into GPU memory without fragmentation — and is now an Apache-2.0 project with roughly 90,000 GitHub stars and more than 2,000 contributors. The documentation is organised as Examples, Features, Inference and Serving, Integrations, Configuration and Models, Design Documents, and a full Python and CLI API reference. The Features section is where most engineers spend their time: quantization (FP8, MXFP8/MXFP4, INT4/INT8, GPTQ, AWQ, GGUF), speculative decoding, prefix caching, structured and guided output, multi-LoRA serving, tool calling, and reasoning-model output parsing. Inference and Serving covers the parallelism axes — tensor, pipeline, data, expert and context parallel — with troubleshooting for multi-node deployments. The Design Documents section is unusually good for an inference project, explaining the V1 engine architecture, CUDA graph capture and MoE kernel design rather than only the public API. Supported hardware spans NVIDIA, AMD and Intel GPUs, x86/ARM/PowerPC CPUs, Google TPUs, Intel Gaudi and Apple Silicon, across more than 200 model architectures. The server exposes an OpenAI-compatible API, which is why vLLM is the default self-hosting target for most enterprise inference stacks.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- Reference documentation, self-paced — about 2-3 hours to work through the serving quickstart and the main feature guides
- Provider
- vLLM Project
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Stand up an OpenAI-compatible inference server in front of an open-weights model
- ✓Choose a quantization scheme — FP8, INT4, GPTQ, AWQ or GGUF — against your accuracy and memory budget
- ✓Pick between tensor, pipeline, data, expert and context parallelism when a model exceeds one GPU
- ✓Configure prefix caching and continuous batching to raise throughput on repeated-prompt workloads
- ✓Serve many fine-tuned variants of one base model at once using multi-LoRA adapters
- ✓Enforce structured JSON or grammar-constrained output directly at the decoding layer
- ✓Read the design documents to understand how PagedAttention and the V1 engine actually schedule requests
Highlights
- •The design documents explain engine internals, not just the API surface — rare among inference projects
- •Covers an unusually wide hardware matrix: NVIDIA, AMD, Intel, TPU, Gaudi, Apple Silicon and CPUs
- •Over 200 supported model architectures, including MoE and multimodal families
- •OpenAI-compatible API means existing client code usually needs only a base-URL change
- •Backed by 2,000+ contributors, so the docs track fast-moving features like speculative decoding closely
Who It’s For
Best For
- ✓Platform engineers moving inference off a hosted API onto owned GPUs
- ✓ML engineers who need to fit a large open-weights model onto specific hardware
- ✓Anyone debugging throughput, latency or out-of-memory failures in a production LLM service
Prerequisites
- •Comfortable with Python, the CLI, and running services in Docker or Kubernetes
- •Basic transformer inference vocabulary — KV cache, batching, context length, quantization
- •Access to a GPU (or CPU backend) to run anything beyond the reading
FAQ
What is vLLM Documentation?
The official documentation for vLLM, the open-source inference and serving engine that introduced PagedAttention. It covers deploying an OpenAI-compatible LLM server, the quantization and parallelism options that decide whether a model fits your GPUs, and the design documents explaining the engine internals. This is the manual you work through when you move from calling a hosted API to running models on your own hardware.
Is vLLM Documentation free?
vLLM Documentation is free to access.
What level is vLLM Documentation for?
vLLM Documentation is aimed at a intermediate audience. Recommended background: Comfortable with Python, the CLI, and running services in Docker or Kubernetes, Basic transformer inference vocabulary — KV cache, batching, context length, quantization, Access to a GPU (or CPU backend) to run anything beyond the reading.
How long does vLLM Documentation take?
Expect roughly Reference documentation, self-paced — about 2-3 hours to work through the serving quickstart and the main feature guides. Most learners work through it at their own pace.
What will I learn from vLLM Documentation?
You'll learn: Stand up an OpenAI-compatible inference server in front of an open-weights model; Choose a quantization scheme — FP8, INT4, GPTQ, AWQ or GGUF — against your accuracy and memory budget; Pick between tensor, pipeline, data, expert and context parallelism when a model exceeds one GPU; Configure prefix caching and continuous batching to raise throughput on repeated-prompt workloads; Serve many fine-tuned variants of one base model at once using multi-LoRA adapters; Enforce structured JSON or grammar-constrained output directly at the decoding layer; Read the design documents to understand how PagedAttention and the V1 engine actually schedule requests.
Topics
Sources
This page was written from 3 sources, 2 on domains other than docs.vllm.ai.