SGLang Documentation
by LMSYS / SGLang Project
The operating manual for serving open-weight LLMs at production scale on your own GPUs.
Overview
SGLang's documentation is the operating manual for a high-performance LLM and multimodal serving framework maintained under the non-profit LMSYS organisation, and it is organised into eight top-level sections: Getting Started (installation, quickstart), Basic Usage (OpenAI-, Anthropic- and Ollama-compatible HTTP APIs, an offline engine API, native endpoints and sampling parameters), Advanced Features, Supported Models, a Developer Guide, References, Hardware Platforms, a per-model Cookbook, and a separate SGLang Diffusion tree. Advanced Features is where the depth is: roughly forty pages covering server arguments, attention backends, HiSparse hierarchical sparse attention, speculative and adaptive speculative decoding, structured outputs and tool/reasoning parsers, quantization and quantized KV cache, expert, pipeline and decode-context parallelism, LoRA serving, prefill-decode and encode-prefill-decode disaggregation, hierarchical KV caching (HiCache) with pluggable storage backends, deterministic inference, observability, and integration into RL training loops. The Hardware Platforms section documents NVIDIA, AMD, Intel Xeon CPU, Apple Silicon Metal, Ascend NPU, TPU, XPU, Jetson Orin and Moore Threads targets, and the Cookbook adds forty-plus per-model recipes plus twenty-plus diffusion guides. The upstream Apache-2.0 repository carries 31.8k GitHub stars and lists xAI, NVIDIA, AMD, LinkedIn, Cursor, Oracle Cloud, Azure and AWS among its adopters.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- Reference documentation; ~3-4 hours for the install -> quickstart -> server-arguments path
- Provider
- LMSYS / SGLang Project
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Launch an OpenAI-compatible SGLang server and tune its server arguments
- ✓Configure RadixAttention prefix caching and hierarchical KV caching with HiCache
- ✓Enable speculative and adaptive speculative decoding to cut generation latency
- ✓Apply FP8, BF16 and NVFP4 quantization plus a quantized KV cache
- ✓Shard large MoE models with expert, pipeline and decode-context parallelism
- ✓Serve constrained JSON and grammar-based structured outputs with reasoning parsers
- ✓Run prefill-decode disaggregation and multi-node Kubernetes or SageMaker deployments
- ✓Serve many LoRA adapters against a single shared base model
Highlights
- •Documents nine hardware targets including Ascend NPU, Google TPU and Apple Metal
- •The cookbook carries per-model recipes for 40+ LLMs and 20+ diffusion models
- •Dedicated pages for deterministic inference and observability, rare in serving docs
- •Backed by a 31.8k-star Apache-2.0 repo used by xAI, LinkedIn, Cursor and Oracle Cloud
- •Covers RL-training integration, so the same engine drives rollouts and production
Who It’s For
Best For
- ✓Inference engineers deploying open-weight LLMs on their own GPUs
- ✓Platform teams sizing multi-node mixture-of-experts serving clusters
- ✓RL engineers wiring a fast rollout engine into a post-training loop
Prerequisites
- •Working Python plus Linux and CUDA command-line experience
- •Familiarity with transformer inference concepts: KV cache, batching, attention
- •Access to at least one NVIDIA, AMD or other supported accelerator
FAQ
What is SGLang Documentation?
The official documentation for SGLang, the high-performance serving framework for large language and multimodal models maintained under the non-profit LMSYS organisation. It is written for inference and platform engineers who have to stand up their own model serving rather than call a hosted API, and it takes you from a single-GPU quickstart through multi-node mixture-of-experts deployments, quantization, speculative decoding and prefill-decode disaggregation.
Is SGLang Documentation free?
SGLang Documentation is free to access.
What level is SGLang Documentation for?
SGLang Documentation is aimed at a advanced audience. Recommended background: Working Python plus Linux and CUDA command-line experience, Familiarity with transformer inference concepts: KV cache, batching, attention, Access to at least one NVIDIA, AMD or other supported accelerator.
How long does SGLang Documentation take?
Expect roughly Reference documentation; ~3-4 hours for the install -> quickstart -> server-arguments path. Most learners work through it at their own pace.
What will I learn from SGLang Documentation?
You'll learn: Launch an OpenAI-compatible SGLang server and tune its server arguments; Configure RadixAttention prefix caching and hierarchical KV caching with HiCache; Enable speculative and adaptive speculative decoding to cut generation latency; Apply FP8, BF16 and NVFP4 quantization plus a quantized KV cache; Shard large MoE models with expert, pipeline and decode-context parallelism; Serve constrained JSON and grammar-based structured outputs with reasoning parsers; Run prefill-decode disaggregation and multi-node Kubernetes or SageMaker deployments; Serve many LoRA adapters against a single shared base model.
Topics
Sources
This page was written from 3 sources, 2 on domains other than docs.sglang.io.