FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving
by FlashInfer-AI
Read the kernels that actually run under vLLM, SGLang and TensorRT-LLM.
Overview
The docs are versioned with the library — 0.6.18 at the time of writing — and split into three layers. Get Started covers installation and the Python package, the command-line interface, logging and tracing, operation-schema extraction via fi_trace, autotuning configuration, and the experimental APIs, along with JIT build progress, cache behaviour and artifact management, which matter because FlashInfer compiles kernels at runtime. The Tutorials layer is the conceptual part: attention states and recursive attention, KV-cache layouts (NHD versus HND, ragged tensors and page tables), JAX integration through TVM FFI, and a Gemma 3 worked example. The API reference then documents the PyTorch surface: prefill, decode and MLA attention; GEMM in BF16, FP8, FP4 and MXFP8; fused MoE across multiple backends; communication primitives including AllReduce and all-to-all; sorting-free sampling with top-k, top-p and min-p; RMSNorm and LayerNorm; FP4/FP8/NVFP4 quantization; and specialized operators such as KDA, Mamba and GDN. Backends can be swapped between cuDNN, CUTLASS, TensorRT-LLM, CuteDSL and PyTorch native, across Turing through Blackwell (SM75 to SM100+). The project has roughly 6.4k GitHub stars and is used by vLLM, SGLang, TensorRT-LLM, TGI, MLC-LLM, LightLLM, lorax and ScaleLLM, and is backed by the paper 'FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving' (arXiv:2501.01005).
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-4 hours for Get Started plus the attention and KV-cache tutorials; ongoing as an API reference
- Provider
- FlashInfer-AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Install FlashInfer and drive it through its CLI, JIT cache and artifact management tools
- ✓Choose a KV-cache layout — NHD versus HND, ragged tensors or page tables — for your serving pattern
- ✓Call prefill, decode and MLA attention kernels through one unified PyTorch API
- ✓Swap attention backends between cuDNN, CUTLASS, TensorRT-LLM, CuteDSL and PyTorch native
- ✓Run fused MoE kernels and FP8/FP4/MXFP8 quantized GEMM on Hopper and Blackwell parts
- ✓Use sorting-free top-k, top-p and min-p sampling instead of a full sort per step
- ✓Trace and autotune kernels with fi_trace, logging and the autotuning configuration pages
Highlights
- •Versioned with the library (0.6.18) rather than being a static README that drifts from the code
- •These are the kernels underneath vLLM, SGLang, TensorRT-LLM, TGI, MLC-LLM, LightLLM and lorax — reading them explains your serving stack's behaviour
- •Tutorials teach the math (attention states, recursive attention, cascading for shared prefixes), not just call signatures
- •Covers Turing through Blackwell (SM75-SM100+), including FP4/NVFP4 paths that only exist on newer silicon
- •Apache-2.0 with ~6.4k stars and an accompanying research paper (arXiv:2501.01005) for the design rationale
Who It’s For
Best For
- ✓Inference engineers tuning vLLM or SGLang throughput, latency and memory per token
- ✓Kernel and performance engineers writing custom attention, MoE or quantized paths
- ✓Teams deploying FP8/FP4 quantized models on Hopper or Blackwell GPUs
- ✓Anyone debugging why a serving engine's numbers differ from the benchmark blog post
Prerequisites
- •Working CUDA and GPU knowledge — memory layout, occupancy, and what a kernel launch costs
- •Comfortable with PyTorch tensors and the mechanics of transformer attention and KV caching
- •Access to a supported NVIDIA GPU (Turing through Blackwell) to run anything hands-on
- •Familiarity with an LLM serving engine such as vLLM, SGLang or TensorRT-LLM to apply it against
FAQ
What is FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving?
The official documentation for FlashInfer, the Apache-2.0 GPU kernel library and kernel generator that provides attention, GEMM, MoE and sampling kernels for LLM inference. It is aimed at inference and performance engineers who need to understand or tune what their serving engine does per token, and at anyone writing custom attention or quantized paths on Hopper and Blackwell hardware rather than accepting framework defaults.
Is FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving free?
FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving is free to access.
What level is FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving for?
FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving is aimed at a advanced audience. Recommended background: Working CUDA and GPU knowledge — memory layout, occupancy, and what a kernel launch costs, Comfortable with PyTorch tensors and the mechanics of transformer attention and KV caching, Access to a supported NVIDIA GPU (Turing through Blackwell) to run anything hands-on, Familiarity with an LLM serving engine such as vLLM, SGLang or TensorRT-LLM to apply it against.
How long does FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving take?
Expect roughly ~3-4 hours for Get Started plus the attention and KV-cache tutorials; ongoing as an API reference. Most learners work through it at their own pace.
What will I learn from FlashInfer Documentation — GPU Kernel Library for LLM Inference Serving?
You'll learn: Install FlashInfer and drive it through its CLI, JIT cache and artifact management tools; Choose a KV-cache layout — NHD versus HND, ragged tensors or page tables — for your serving pattern; Call prefill, decode and MLA attention kernels through one unified PyTorch API; Swap attention backends between cuDNN, CUTLASS, TensorRT-LLM, CuteDSL and PyTorch native; Run fused MoE kernels and FP8/FP4/MXFP8 quantized GEMM on Hopper and Blackwell parts; Use sorting-free top-k, top-p and min-p sampling instead of a full sort per step; Trace and autotune kernels with fi_trace, logging and the autotuning configuration pages.
Topics
Sources
This page was written from 2 sources, 1 on domains other than docs.flashinfer.ai.