XGrammar Documentation — Fast, Portable Structured Generation for LLMs
by MLC AI
Learn how constrained decoding actually works, from the engine vLLM, SGLang and TensorRT-LLM already use for structured output.
Overview
The docs (version 0.2.5) are laid out as Get Started (Installation, Quick Start, Constrained Decoding, Workflow of XGrammar), Structural Tag (Structural Tag Usage, Tool Calling and Reasoning, Advanced Topics), Using XGrammar (Integration with LLM Engine, Serialization, Runtime Safeguards, JavaScript API), Defining Structures (JSON Generation, EBNF Grammar, Lark Grammar), a Developer Guide, and a Python API reference covering Grammar, TokenizerInfo, GrammarCompiler, CompiledGrammar, GrammarMatcher, Structural Tag, Builtin Structural Tag, the OpenAI Tool Call Schema, bitmask operations, config and exceptions. XGrammar is a C++ core with Python, C++, JavaScript and Swift APIs, running on Linux, macOS and Windows across CPU, NVIDIA and AMD GPUs, Apple Silicon and TPU. It is the default structured-generation backend for vLLM, SGLang, TensorRT-LLM and MLC-LLM, and is also integrated into OpenVINO GenAI, Modular's MAX and Mirai, which is why its behaviour shows up in production stacks whether or not you called it directly. The project is Apache-2.0 with roughly 1.9k GitHub stars and about 528 commits, maintained by the MLC AI team. Its design is published: XGrammar-2 (arXiv 2601.04426, ACM CAIS '26) adds TagDispatch for tag-triggered structure switching mid-generation and a Cross-Grammar Cache for substructure-level reuse across grammars, plus an Earley-based adaptive token mask cache, JIT compilation and repetition state compression, claiming over 6x faster grammar compilation and near-zero end-to-end serving overhead.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-4 hours for Get Started, Structural Tag and the engine-integration guides; self-paced
- Provider
- MLC AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Guarantee valid JSON output by compiling a JSON Schema into a per-token mask
- ✓Write EBNF and Lark grammars for formats a JSON Schema cannot express
- ✓Use structural tags to switch grammars mid-generation for tool calls and reasoning traces
- ✓Integrate a GrammarMatcher and bitmask operations into a custom inference loop
- ✓Understand how constrained decoding masks logits without destroying throughput
- ✓Serialize and cache compiled grammars to keep compilation off the request hot path
- ✓Apply runtime safeguards so a malformed grammar cannot stall a serving process
- ✓Map OpenAI tool-call schemas onto the built-in structural tag support
Highlights
- •This is the engine underneath vLLM, SGLang, TensorRT-LLM and MLC-LLM — reading it explains structured-output behaviour you already depend on
- •C++ core with Python, C++, JavaScript and Swift APIs across Linux, macOS, Windows, NVIDIA and AMD GPUs, Apple Silicon and TPU
- •Structural tags are built for agentic workloads where the required structure changes within a single response, not just across requests
- •The design is peer-reviewed: XGrammar-2 appeared at ACM CAIS '26 claiming over 6x faster compilation and near-zero serving overhead
- •Documents serialization and runtime safeguards — the operational details most structured-output tutorials leave out
Who It’s For
Best For
- ✓Engineers running their own vLLM or SGLang deployment who need reliable structured output
- ✓Agent developers debugging flaky or malformed tool-call formatting
- ✓Inference and serving infrastructure developers integrating constrained decoding
Prerequisites
- •Understanding of tokenization and how logits are sampled at each decoding step
- •Python (or C++) plus hands-on experience with an inference engine such as vLLM or Transformers
- •Familiarity with JSON Schema; EBNF or context-free grammar concepts help for the grammar sections
FAQ
What is XGrammar Documentation — Fast, Portable Structured Generation for LLMs?
Official documentation for XGrammar, the Apache-2.0 constrained-decoding library that guarantees an LLM's output matches a JSON Schema, a regex, or a custom context-free grammar. It is the default structured-generation backend inside vLLM, SGLang, TensorRT-LLM and MLC-LLM, so this is where you learn what your serving stack is already doing on every request. After it you can enforce schemas, debug tool-calling grammars, and wire structured output into your own inference loop.
Is XGrammar Documentation — Fast, Portable Structured Generation for LLMs free?
XGrammar Documentation — Fast, Portable Structured Generation for LLMs is free to access.
What level is XGrammar Documentation — Fast, Portable Structured Generation for LLMs for?
XGrammar Documentation — Fast, Portable Structured Generation for LLMs is aimed at a intermediate audience. Recommended background: Understanding of tokenization and how logits are sampled at each decoding step, Python (or C++) plus hands-on experience with an inference engine such as vLLM or Transformers, Familiarity with JSON Schema; EBNF or context-free grammar concepts help for the grammar sections.
How long does XGrammar Documentation — Fast, Portable Structured Generation for LLMs take?
Expect roughly ~3-4 hours for Get Started, Structural Tag and the engine-integration guides; self-paced. Most learners work through it at their own pace.
What will I learn from XGrammar Documentation — Fast, Portable Structured Generation for LLMs?
You'll learn: Guarantee valid JSON output by compiling a JSON Schema into a per-token mask; Write EBNF and Lark grammars for formats a JSON Schema cannot express; Use structural tags to switch grammars mid-generation for tool calls and reasoning traces; Integrate a GrammarMatcher and bitmask operations into a custom inference loop; Understand how constrained decoding masks logits without destroying throughput; Serialize and cache compiled grammars to keep compilation off the request hot path; Apply runtime safeguards so a malformed grammar cannot stall a serving process; Map OpenAI tool-call schemas onto the built-in structural tag support.
Topics
Sources
This page was written from 3 sources, 2 on domains other than xgrammar.mlc.ai.