LLM Compressor Documentation

by vLLM Project

AdvancedDocumentationFree~1-2 hours to run a first FP8 recipe, then ongoing reference

The official recipe book for quantizing an LLM into a format vLLM serves — FP8, INT4, NVFP4 and MXFP4.

Start LearningAdded Aug 12, 2026 · Updated Aug 12, 2026

Overview

LLM Compressor is maintained by the vLLM project (Apache-2.0, ~3.7k GitHub stars, over 3,100 commits on main) and is the reference path from a Hugging Face checkpoint to a compressed-tensors artifact vLLM loads directly. The docs are split into Getting Started, compression guides (schemes and observers), key-model pages, deployment, performance, and developer/API reference. The central abstraction is a recipe: you declare a QuantizationModifier with a target module pattern and a scheme, call oneshot(model=..., recipe=...), then save_pretrained(). Documented algorithms include round-to-nearest as the baseline, GPTQ and AWQ for calibration-based weight quantization, SmoothQuant for activation-outlier mitigation ahead of W8A8, rotation-based SpinQuant and QuIP, AutoRound, REAP expert pruning for mixture-of-experts models, and FP8/NVFP4 KV-cache and attention quantization; version 0.12.0 adds Transformers v5 integration, multi-GPU model-free PTQ, and support for stacking multiple modifiers so AWQ and GPTQ can run in a single pass. Supported schemes span weight-only W4A16 and W8A16, weight-and-activation W8A8 in INT8 and FP8, W4AFP8 and W4AINT8 hybrids, and the microscale MXFP8, MXFP4 and NVFP4 formats targeting Blackwell-class GPUs. Examples are organised by scheme (quantization_w8a8_fp8, W4A16, KV cache), by model type (multimodal_vision, multimodal_audio, quantizing_moe) and by algorithm (awq, autoround), each scheme carrying at least one Llama reference example; the docs recommend the data-free w8a8_fp8 path first. Dedicated pages cover Llama 4, Qwen 3.5/3.6, Gemma 4 and Mistral Large 3. Note that 2:4 sparsity has been dropped for lack of hardware support.

At a Glance

Topic: Fine-Tuning
Level: Advanced
Format: Documentation
Cost: Free
Duration: ~1-2 hours to run a first FP8 recipe, then ongoing reference
Provider: vLLM Project
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓Write an llmcompressor recipe and run oneshot post-training quantization on a checkpoint
✓Choose between W4A16, W8A8-INT8, FP8 and NVFP4 based on your target GPU
✓Apply GPTQ and AWQ with calibration data, and stack multiple modifiers in one pass
✓Use SmoothQuant to tame activation outliers before W8A8 quantization
✓Quantize the KV cache and attention to cut long-context memory cost
✓Compress mixture-of-experts and multimodal vision/audio models with dedicated recipes
✓Load the resulting compressed-tensors model in vLLM and validate accuracy

Highlights

•Maintained inside the vLLM project, so the output format is guaranteed servable
•Covers the microscale FP4 formats (NVFP4, MXFP4) that most quantization guides omit
•Every scheme ships a runnable Llama example rather than prose-only guidance
•Documents hardware requirements and memory cost per scheme, not only accuracy
•Honest about scope — 2:4 sparsity was removed rather than left rotting in the docs

Who It’s For

Best For

✓Inference engineers cutting GPU memory or serving cost for a self-hosted model
✓Teams standardising on vLLM who need quantized checkpoints they control
✓Engineers comparing FP8 versus INT4 accuracy tradeoffs on their own evals

Prerequisites

•Python plus working familiarity with Hugging Face transformers and model checkpoints
•A suitable GPU and prior experience serving a model with vLLM
•Conceptual grounding in quantization — weights vs activations, calibration data

FAQ

What is LLM Compressor Documentation?

LLM Compressor is the vLLM project's Transformers-compatible library for post-training compression, and these docs are how you turn a Hugging Face checkpoint into a quantized model vLLM loads at a fraction of the memory. They document the algorithms (GPTQ, AWQ, SmoothQuant, SpinQuant, AutoRound), every supported weight and activation scheme, and runnable end-to-end examples. Work through one and you can ship an FP8 or INT4 model yourself.

Is LLM Compressor Documentation free?

LLM Compressor Documentation is free to access.

What level is LLM Compressor Documentation for?

LLM Compressor Documentation is aimed at a advanced audience. Recommended background: Python plus working familiarity with Hugging Face transformers and model checkpoints, A suitable GPU and prior experience serving a model with vLLM, Conceptual grounding in quantization — weights vs activations, calibration data.

How long does LLM Compressor Documentation take?

Expect roughly ~1-2 hours to run a first FP8 recipe, then ongoing reference. Most learners work through it at their own pace.

What will I learn from LLM Compressor Documentation?

You'll learn: Write an llmcompressor recipe and run oneshot post-training quantization on a checkpoint; Choose between W4A16, W8A8-INT8, FP8 and NVFP4 based on your target GPU; Apply GPTQ and AWQ with calibration data, and stack multiple modifiers in one pass; Use SmoothQuant to tame activation outliers before W8A8 quantization; Quantize the KV cache and attention to cut long-context memory cost; Compress mixture-of-experts and multimodal vision/audio models with dedicated recipes; Load the resulting compressed-tensors model in vLLM and validate accuracy.

Topics

quantizationvllmfp8awqgptqinference-optimization

Sources

This page was written from 3 sources, 1 on domains other than docs.vllm.ai.