ModelsFine-TuningFrameworks

Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM

by vLLM Project (Red Hat)

AdvancedDocumentationFree~1 hour to serve a pre-trained speculator; a day or more to train one end to end, depending on GPU count and dataset size (estimate)

Train EAGLE-3, DFlash, P-EAGLE, DSpark or MTP draft models for your own target model and serve them in vLLM with one command.

Start LearningAdded Oct 15, 2026 · Updated Oct 15, 2026

Overview

Speculative decoding has a small draft model, the speculator, propose tokens that the larger target model verifies in one forward pass, so accepted tokens follow the target model's own distribution. Speculators packages the whole workflow: offline or online generation of the target model's hidden states with vLLM, training of single and multi-layer drafters for dense, MoE and vision-language models, a Hugging Face-compatible checkpoint format with converters for research checkpoints, and direct loading in vLLM via vllm serve. The docs cover an overview, supported models, installation, a user guide, six tutorials (Train a Speculator, Multi-Node Training over a Mooncake backend, Response Regeneration, Evaluating Model Performance, Throughput vs. Interactivity, Serve in vLLM), and API and CLI references. The main tutorial walks through speculators prepare-data, launching vLLM, generate-offline-data, torchrun training with per-algorithm hyperparameters (EAGLE-3, P-EAGLE, DFlash, experimental DFlash2, DSpark and MTP), and serving checkpoint_best. It is candid about cost and results: offline hidden states run about 268 MB per sample (about 1.3 TB for 5K samples) in its example, and its 5K-sample runs are labeled sanity checks rather than benchmarks. The team has trained drafters for Llama, Qwen3, gpt-oss, Gemma 4, Mistral Small 4 and Nemotron 3 targets. Version 0.8.0 shipped on PyPI on September 3, 2026 and 0.8.1 on October 10, 2026.

At a Glance

Topic: Models
Level: Advanced
Format: Documentation
Cost: Free
Duration: ~1 hour to serve a pre-trained speculator; a day or more to train one end to end, depending on GPU count and dataset size (estimate)
Provider: vLLM Project (Red Hat)
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓Serve a pre-trained speculator alongside its verifier with a single vllm serve command
✓Regenerate dataset responses with your target model so the drafter trains on on-policy data
✓Prepare training data with speculators prepare-data, including the token frequencies for a reduced draft vocabulary
✓Choose between online hidden-state generation and offline caching based on disk and GPU budget
✓Train EAGLE-3, P-EAGLE, DFlash, DSpark or MTP drafters with torchrun and the documented hyperparameters
✓Stream hidden states between extraction and training nodes for targets too large for one node
✓Read acceptance rate and acceptance length from vLLM logs and benchmark throughput against per-user speed
✓Convert third-party research checkpoints into the standardized Speculators format for vLLM

Highlights

•One library covers six drafting methods (EAGLE-3, P-EAGLE, DFlash, DFlash2, DSpark and MTP fine-tuning) behind the same data and training pipeline
•Ready-made drafters exist for large targets such as Qwen3-235B, gpt-oss-120b and Nemotron 3 Ultra 550B-A55B
•Honest example numbers: it reports DFlash acceptance length of 1.47 on a 5K-sample run and tells you that is a sanity check, with roughly 40% first-token acceptance as the typical figure
•Actively released: 0.8.0 in September and 0.8.1 in October 2026; the GitHub repo has 883 stars and 256 forks under Apache-2.0
•A Throughput vs. Interactivity tutorial measures the real trade-off on live prompts, following the InferenceX method

Who It’s For

Best For

✓Inference engineers who serve a model in vLLM and want lower per-token latency without changing outputs
✓Teams fine-tuning or self-hosting open-weight models who need a drafter matched to their checkpoint
✓ML engineers comparing speculative decoding methods on their own traffic
✓Platform teams tuning throughput versus interactivity for chat workloads

Prerequisites

•Experience serving models with vLLM and reading its metrics
•PyTorch distributed training basics (torchrun, multi-GPU) and a GPU node; the main tutorial example assumes four GPUs
•Python 3.10+ on Linux or macOS, plus substantial disk if you use offline hidden-state mode

FAQ

What is Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM?

The official documentation for Speculators, the vLLM project's library for building, training and storing speculative decoding draft models. It is for inference engineers who want lower latency from a model they already serve in vLLM. After the tutorials you can generate on-policy training data, train a drafter, measure acceptance and deploy it.

Is Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM free?

Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM is free to access.

What level is Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM for?

Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM is aimed at a advanced audience. Recommended background: Experience serving models with vLLM and reading its metrics, PyTorch distributed training basics (torchrun, multi-GPU) and a GPU node; the main tutorial example assumes four GPUs, Python 3.10+ on Linux or macOS, plus substantial disk if you use offline hidden-state mode.

How long does Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM take?

Expect roughly ~1 hour to serve a pre-trained speculator; a day or more to train one end to end, depending on GPU count and dataset size (estimate). Most learners work through it at their own pace.

What will I learn from Speculators Documentation — Train and Serve Speculative Decoding Draft Models for vLLM?

You'll learn: Serve a pre-trained speculator alongside its verifier with a single vllm serve command; Regenerate dataset responses with your target model so the drafter trains on on-policy data; Prepare training data with speculators prepare-data, including the token frequencies for a reduced draft vocabulary; Choose between online hidden-state generation and offline caching based on disk and GPU budget; Train EAGLE-3, P-EAGLE, DFlash, DSpark or MTP drafters with torchrun and the documented hyperparameters; Stream hidden states between extraction and training nodes for targets too large for one node; Read acceptance rate and acceptance length from vLLM logs and benchmark throughput against per-user speed; Convert third-party research checkpoints into the standardized Speculators format for vLLM.

Topics

speculative-decodingvllmeagle-3inference-optimizationdraft-modelsllm-serving

Sources

This page was written from 6 sources, 2 on domains other than docs.vllm.ai.