Hands-On LLM Serving and Optimization: Hosting LLMs at Scale
by O'Reilly Media (Chi Wang, Peiheng Hu)
Tune and benchmark LLM serving with vLLM, SGLang, TensorRT-LLM and llama.cpp, using chapter-by-chapter notebooks.
Overview
Hands-On LLM Serving and Optimization: Hosting LLMs at Scale is an O'Reilly book by Chi Wang and Peiheng Hu, listed at 374 pages with an ebook release date of April 28, 2026 on Helion. It has ten chapters. Chapters 1 to 4 cover model serving basics, transformer internals, the KV cache, prefill and decode, single-model and multi-model service design, NVIDIA Triton, and build-or-buy choices for agentic and RAG workloads. Chapter 5 sizes the problem through GPU specs, model-loading bottlenecks, KV cache sizing and arithmetic-intensity analysis. Chapters 6 and 7 teach dynamic and continuous batching, chunked prefill, scalable attention, kernel fusion, quantization, distillation, pruning, prefix caching, RadixAttention, speculative decoding, multi-GPU and multinode parallelism, and prefill-decode disaggregation. Chapter 8 compares vLLM, TensorRT-LLM, SGLang and llama.cpp. Chapter 9 walks through optimizing Qwen3-14B with vLLM, from hardware inspection to distributed serving. Chapter 10 covers semantic caching, profiling, multimodal serving, multi-LoRA serving and serving for reinforcement learning. The companion GitHub repository organizes notebooks by chapter. Chapters 1 and 5 have no code, and many notebooks need an NVIDIA GPU, with Google Colab's GPU runtime as the suggested alternative. On November 4, 2026 the repository showed 147 stars, 58 forks and 95 commits, and its page named no open-source license. Helion's listing describes Wang as Director of Engineering at Einstein AI at Salesforce with over 18 years in AI and distributed systems, and Hu as an LLM inference engineer at NVIDIA who previously worked at Salesforce and Microsoft Azure.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Book
- Cost
- Paid
- Duration
- 374 pages plus chapter notebooks, self-paced
- Provider
- O'Reilly Media (Chi Wang, Peiheng Hu)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Explain autoregressive generation, attention, the KV cache, and the prefill and decode phases
- ✓Build single-model and multi-model serving services, including an NVIDIA Triton example
- ✓Estimate GPU requirements, model-loading bottlenecks and KV cache size before you deploy a model
- ✓Apply continuous batching, chunked prefill, quantization, distillation, pruning and prefix caching to serving workloads
- ✓Use speculative decoding, multinode parallelism and prefill-decode disaggregation to serve larger models and longer contexts
- ✓Compare vLLM, TensorRT-LLM, SGLang and llama.cpp internals, then choose a framework for your workload
- ✓Follow a step-by-step plan that benchmarks Qwen3-14B on vLLM through to distributed serving
Highlights
- •Chapter 9 works through one model, Qwen3-14B, from hardware inspection to distributed serving and its trade-offs
- •The GitHub notebooks follow the book's chapters and include a from-scratch serving service plus LMCache and speculative decoding examples
- •Chapter 8 goes inside vLLM and covers three other serving frameworks, so the comparison spans more than one vendor's stack
- •Chapter 10 reaches newer serving cases such as multi-LoRA, multimodal workloads and serving for reinforcement learning
- •Budget for GPU access: many notebooks need an NVIDIA GPU or a Colab GPU runtime, and model weights may need a separate Git LFS pull
Who It’s For
Best For
- ✓Engineers building or operating LLM inference infrastructure
- ✓Application and agent teams that need to understand latency, throughput and serving cost
- ✓ML engineers moving from model training into production inference
Prerequisites
- •Comfort running Python notebooks and installing dependencies in virtual environments
- •Access to an NVIDIA GPU or a Google Colab GPU runtime for many of the examples
FAQ
What is Hands-On LLM Serving and Optimization: Hosting LLMs at Scale?
Hands-On LLM Serving and Optimization is an O'Reilly book for engineers who run LLM inference infrastructure and developers whose applications depend on it. After reading it, you can size GPUs and KV cache, apply batching, quantization and speculative decoding, compare serving frameworks, and benchmark a Qwen3-14B deployment on vLLM.
Is Hands-On LLM Serving and Optimization: Hosting LLMs at Scale free?
Hands-On LLM Serving and Optimization: Hosting LLMs at Scale is a paid resource.
What level is Hands-On LLM Serving and Optimization: Hosting LLMs at Scale for?
Hands-On LLM Serving and Optimization: Hosting LLMs at Scale is aimed at a advanced audience. Recommended background: Comfort running Python notebooks and installing dependencies in virtual environments, Access to an NVIDIA GPU or a Google Colab GPU runtime for many of the examples.
How long does Hands-On LLM Serving and Optimization: Hosting LLMs at Scale take?
Expect roughly 374 pages plus chapter notebooks, self-paced. Most learners work through it at their own pace.
What will I learn from Hands-On LLM Serving and Optimization: Hosting LLMs at Scale?
You'll learn: Explain autoregressive generation, attention, the KV cache, and the prefill and decode phases; Build single-model and multi-model serving services, including an NVIDIA Triton example; Estimate GPU requirements, model-loading bottlenecks and KV cache size before you deploy a model; Apply continuous batching, chunked prefill, quantization, distillation, pruning and prefix caching to serving workloads; Use speculative decoding, multinode parallelism and prefill-decode disaggregation to serve larger models and longer contexts; Compare vLLM, TensorRT-LLM, SGLang and llama.cpp internals, then choose a framework for your workload; Follow a step-by-step plan that benchmarks Qwen3-14B on vLLM through to distributed serving.
Topics
Sources
This page was written from 3 sources, 2 on domains other than orca3.github.io.