TensorRT LLM Documentation
by NVIDIA
NVIDIA's own inference stack, documented down to disaggregated serving and speculative decoding.
Overview
TensorRT LLM is NVIDIA's inference stack for serving large language models on its own GPUs, and this site is its primary reference — the build inspected was last updated 30 July 2026 against the 1.3.0 release line. It opens with Getting Started (overview, quick start, pip and container installation, supported hardware), then a Deployment Guide with LLM API examples covering text generation, async and streaming inference and distributed multi-GPU runs, plus per-model deployment recipes and online serving through trtllm-serve. A Models section carries the supported-model feature matrix and instructions for adding a new architecture. CLI tooling is documented separately: trtllm-bench for benchmarking, trtllm-eval for evaluation, trtllm-serve for production serving. The API reference covers the high-level Python LLM API for offline inference alongside the C++ runtime and its configuration classes. The Features chapters are where the engineering value sits: attention variants (MHA, MQA, GQA), quantization, speculative decoding (documented at up to 3.6x throughput), disaggregated prefill/decode serving, tensor, pipeline and expert parallelism, LoRA adapters, multimodal support and KV cache optimization. The project is Apache 2.0 with roughly 14.3k GitHub stars, is now architected on PyTorch, and includes AutoDeploy for moving PyTorch models over without hand-building engines. Hardware coverage centers on H100, H200, L4, B200 and Blackwell.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- Reference docs; ~2-3 hours to work through the quick start and deployment guide
- Provider
- NVIDIA
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Serve a large language model on NVIDIA GPUs using the Python LLM API
- ✓Stand up an online inference endpoint with trtllm-serve for production traffic
- ✓Apply quantization recipes to cut memory footprint and raise throughput
- ✓Configure speculative decoding, documented at up to 3.6x throughput improvement
- ✓Split prefill and decode phases across systems using disaggregated serving
- ✓Choose between tensor, pipeline and expert parallelism for multi-GPU deployments
- ✓Benchmark and evaluate a deployment using trtllm-bench and trtllm-eval
Highlights
- •The NVIDIA-native counterpart to vLLM and SGLang — independent comparisons favor it on standardized NVIDIA clusters where latency matters, at the cost of hardware lock-in
- •Now architected on PyTorch with an AutoDeploy path, so the old hand-built-engine workflow is no longer the only entry point
- •Documents frontier serving techniques — disaggregated prefill/decode, expert parallelism, speculative decoding — rather than stopping at a quickstart
- •Apache 2.0 with ~14.3k stars and heavy ongoing activity, so the docs track releases closely
- •Ships benchmarking and evaluation CLIs, so its performance claims can be checked on your own hardware
Who It’s For
Best For
- ✓Inference engineers optimizing latency on NVIDIA H100, H200 or Blackwell fleets
- ✓Platform teams running self-hosted LLM serving at scale
- ✓ML engineers evaluating TensorRT LLM against vLLM or SGLang
- ✓Teams deploying quantized or multimodal models on NVIDIA hardware
Prerequisites
- •Comfortable with Python and Linux/CUDA GPU environments
- •Understanding of LLM inference concepts: KV cache, batching, quantization
- •Access to a supported NVIDIA GPU (Hopper, Ada or Blackwell class)
FAQ
What is TensorRT LLM Documentation?
The official reference for TensorRT LLM, NVIDIA's inference framework for serving large language models on its GPUs. Covers installation, the Python LLM API, production serving via trtllm-serve, quantization, speculative decoding, disaggregated prefill/decode, and multi-GPU parallelism. This is the NVIDIA-native counterpart to vLLM and SGLang for teams standardized on H100, H200 or Blackwell hardware.
Is TensorRT LLM Documentation free?
TensorRT LLM Documentation is free to access.
What level is TensorRT LLM Documentation for?
TensorRT LLM Documentation is aimed at a advanced audience. Recommended background: Comfortable with Python and Linux/CUDA GPU environments, Understanding of LLM inference concepts: KV cache, batching, quantization, Access to a supported NVIDIA GPU (Hopper, Ada or Blackwell class).
How long does TensorRT LLM Documentation take?
Expect roughly Reference docs; ~2-3 hours to work through the quick start and deployment guide. Most learners work through it at their own pace.
What will I learn from TensorRT LLM Documentation?
You'll learn: Serve a large language model on NVIDIA GPUs using the Python LLM API; Stand up an online inference endpoint with trtllm-serve for production traffic; Apply quantization recipes to cut memory footprint and raise throughput; Configure speculative decoding, documented at up to 3.6x throughput improvement; Split prefill and decode phases across systems using disaggregated serving; Choose between tensor, pipeline and expert parallelism for multi-GPU deployments; Benchmark and evaluate a deployment using trtllm-bench and trtllm-eval.
Topics
Sources
This page was written from 3 sources, 2 on domains other than nvidia.github.io.