LLM Inference Handbook
by Modular (originally BentoML)
One structured, open-source reference for serving LLMs fast, cheaply and reliably in production.
Overview
The LLM Inference Handbook is a free, open-source reference that consolidates what engineers need to run large language models in production, knowledge that is otherwise scattered across papers, vendor blogs and GitHub issues. It was created by the BentoML team, launched on Hacker News in July 2025 (366 points), and moved to handbook.modular.com after BentoML joined Modular, which now maintains it; the GitHub repository has 560+ commits with updates as recent as September 2026, and its documentation is licensed CC BY 4.0. The content is organised into seven sections. Foundations covers what LLM inference is, training versus inference, how an LLM generates tokens, where inference runs, and key metrics. Planning your deployment compares serverless and self-hosted inference and walks through choosing a model, choosing a GPU, calculating GPU memory for serving, choosing an inference framework, bring-your-own-cloud and on-prem deployments. Model preparation covers fine-tuning, quantization and distillation. Model interaction explains OpenAI- and Anthropic-compatible APIs, function calling, structured outputs, the Model Context Protocol, prompt engineering, agent harnesses and inference parameters. Inference optimization covers performance benchmarks, static, dynamic and continuous batching, PagedAttention, speculative decoding, prefill-decode disaggregation, prefix caching, inference routing, KV cache offloading, data, tensor, pipeline and expert parallelism, and offline batch inference. Kernel optimization covers GPU architecture, choosing kernel optimization tools and FlashAttention, and Infra and operations covers distributed inference, observability, fast scaling, build and maintenance cost, multi-model pipelines, multi-cloud inference and InferenceOps. More than 18 interactive calculators and simulators accompany the text.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- Self-paced reference, ~45 topic pages across 7 sections; read end to end over several sittings or dip in per topic
- Provider
- Modular (originally BentoML)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Define and measure TTFT, TPOT, end-to-end latency and SLO-based goodput for LLM serving
- ✓Decide between serverless LLM APIs, self-hosted, bring-your-own-cloud and on-prem deployments
- ✓Choose a GPU and calculate the memory required to serve a given model
- ✓Compare fine-tuning, quantization and distillation as ways to prepare a model for inference
- ✓Apply continuous batching, PagedAttention, prefix caching and speculative decoding to reduce latency and cost
- ✓Understand prefill-decode disaggregation, KV cache offloading and data, tensor, pipeline and expert parallelism
- ✓Plan observability, fast scaling and cost control for multi-model and multi-cloud inference
Highlights
- •Consolidates inference knowledge scattered across papers, vendor blogs and GitHub issues into one structured reference
- •18+ interactive tools, including a GPU memory calculator, KV cache memory calculator, batching strategy simulator and latency metrics playground
- •Actively maintained: 560+ commits, with the KV cache offloading and GPU memory pages updated in September 2026
- •Well received on Hacker News at launch (366 points, July 2025) and open source, with CC BY 4.0 documentation
- •Vendor-maintained by Modular, and recent commits add mentions of its MAX platform, so read tool recommendations with that in mind
Who It’s For
Best For
- ✓ML platform and infrastructure engineers deploying open-weight LLMs
- ✓AI engineers choosing between hosted APIs and self-hosting
- ✓Engineering leads estimating GPU capacity and inference cost
- ✓Developers using vLLM or SGLang who want to understand the optimizations underneath
Prerequisites
- •Working knowledge of how transformer LLMs generate text token by token
- •Basic familiarity with GPUs and cloud deployment
- •Some experience calling LLM HTTP APIs helps for the model interaction section
FAQ
What is LLM Inference Handbook?
A free, open-source handbook for engineers who deploy and operate LLMs in production. It explains inference fundamentals, GPU and framework selection, quantization, and optimizations such as continuous batching, prefix caching and speculative decoding, so you can size hardware, cut latency and cost, and reason about serving trade-offs before you commit to an inference stack.
Is LLM Inference Handbook free?
LLM Inference Handbook is free to access.
What level is LLM Inference Handbook for?
LLM Inference Handbook is aimed at a intermediate audience. Recommended background: Working knowledge of how transformer LLMs generate text token by token, Basic familiarity with GPUs and cloud deployment, Some experience calling LLM HTTP APIs helps for the model interaction section.
How long does LLM Inference Handbook take?
Expect roughly Self-paced reference, ~45 topic pages across 7 sections; read end to end over several sittings or dip in per topic. Most learners work through it at their own pace.
What will I learn from LLM Inference Handbook?
You'll learn: Define and measure TTFT, TPOT, end-to-end latency and SLO-based goodput for LLM serving; Decide between serverless LLM APIs, self-hosted, bring-your-own-cloud and on-prem deployments; Choose a GPU and calculate the memory required to serve a given model; Compare fine-tuning, quantization and distillation as ways to prepare a model for inference; Apply continuous batching, PagedAttention, prefix caching and speculative decoding to reduce latency and cost; Understand prefill-decode disaggregation, KV cache offloading and data, tensor, pipeline and expert parallelism; Plan observability, fast scaling and cost control for multi-model and multi-cloud inference.
Topics
Sources
This page was written from 7 sources, 3 on domains other than handbook.modular.com.