Inference Engineering
by Philip Kiely (Baseten Books)
A free, 256-page map of the LLM inference stack, from GPU memory and the KV cache to vLLM, quantization, speculative decoding and autoscaling.
Overview
Inference Engineering (Baseten Books, 2026) is a 256-page guide to serving generative AI models in production, written by Philip Kiely, who joined Baseten in January 2022 and has spoken on inference at NVIDIA GTC, PyTorch Conference, AI Engineer World's Fair and AWS re:Invent. He wrote it from four years of production work at Baseten plus interviews with its engineers. The full text is free to read online at baseten.co, and there is also a paid paperback (Goodreads lists an April 7, 2026 paperback) and a PDF. It is organised into eight chapters. Chapter 0 frames the three layers of inference. Chapter 1 (Prerequisites) covers shared APIs versus dedicated deployments, latency budgets and unit economics, choosing a model and using fine-tuning or distillation, and measuring time-to-first-token and tokens per second. Chapter 2 (Models) walks through prefill and decode, attention, mixture-of-experts and diffusion pipelines. It then uses the ops:byte ratio and arithmetic intensity to show whether a workload is limited by compute or by memory. Chapter 3 (Hardware) covers GPU architecture from HBM down to registers, the Hopper, Blackwell and Rubin generations, NVLink and InfiniBand, multi-instance GPU (MIG), AMD GPUs, TPUs, and running models locally. Chapter 4 (Software) moves from CUDA and PyTorch up to vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo, plus benchmarking and load testing. Chapter 5 (Techniques) covers FP8/FP4 quantization, speculative decoding, KV and prefix caching with cache-aware routing, tensor/pipeline/expert/data parallelism, and splitting prefill and decode onto separate workers. Chapter 6 covers vision-language models, embeddings, speech recognition, text-to-speech, and image and video generation. Chapter 7 covers containerization, autoscaling, multi-cloud GPU capacity, testing and rollout, and client code, and closes with a Baseten-specific section. The appendices hold a glossary and a recommended-reading list.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Book
- Cost
- Free
- Duration
- ~256 pages (~46,000 words); ~4-5 hours to read, a weekend with the calculators and exercises
- Provider
- Philip Kiely (Baseten Books)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Decide between shared model APIs and dedicated deployments using latency budgets, traffic patterns and unit economics
- ✓Measure time-to-first-token and tokens-per-second correctly before you start optimizing any deployment
- ✓Use arithmetic intensity and the ops:byte ratio to tell compute-bound from memory-bound inference workloads
- ✓Compare GPU generations from Hopper to Blackwell and Rubin, plus AMD GPUs, TPUs and multi-GPU interconnects
- ✓Place vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo in the serving stack and load-test them realistically
- ✓Apply FP8/FP4 quantization, speculative decoding, prefix caching and cache-aware routing while weighing their tradeoffs
- ✓Choose tensor, pipeline, expert or data parallelism, and know when prefill-decode disaggregation pays off
- ✓Serve embeddings, speech recognition, text-to-speech, vision-language, image and video models, each with its own bottlenecks
- ✓Run production inference with containerization, autoscaling, multi-cloud GPU capacity, safe rollouts and resilient client code
Highlights
- •Covers the whole inference stack in one book (models, GPUs, engines, optimization techniques and production operations) instead of a single framework
- •Free to read in full online with no login, with a paid paperback; Goodreads shows 4.17/5 from 30 ratings (checked 2026-09-22)
- •Goes beyond text, with a full chapter on serving embedding, speech (ASR/TTS), vision-language, image and video models
- •Built on four years of Baseten production work and interviews with its engineers; Scott Hanselman's Hanselminutes podcast covered it at launch (Feb 26, 2026)
- •Honest caveat: it is a breadth-first overview from a vendor. Reviewers call it a 'bird's eye view', and the last section is Baseten-specific, so pair it with engine docs for depth
Who It’s For
Best For
- ✓Backend and platform engineers moving from calling hosted model APIs to self-hosting open-weight models
- ✓ML engineers tasked with cutting inference latency or GPU cost on an existing deployment
- ✓Infrastructure architects choosing GPUs, inference engines and serving topologies
- ✓Technical leads deciding whether dedicated inference deployments are worth the operational cost
Prerequisites
- •A working software-engineering background and comfort with APIs, latency and throughput concepts
- •Basic familiarity with what LLMs and transformers are; the book does not teach machine learning from zero
- •No CUDA experience needed. The book notes that most inference work today happens at the inference-engine level
FAQ
What is Inference Engineering?
Inference Engineering is a free online book by Baseten's Philip Kiely for software and ML engineers who need to serve LLMs and other generative models in production. It explains why inference is slow or expensive and how to fix it, so you can choose GPUs, pick an inference engine, apply optimizations like quantization and speculative decoding, and run autoscaled deployments with confidence.
Is Inference Engineering free?
Inference Engineering is free to access.
What level is Inference Engineering for?
Inference Engineering is aimed at a intermediate audience. Recommended background: A working software-engineering background and comfort with APIs, latency and throughput concepts, Basic familiarity with what LLMs and transformers are; the book does not teach machine learning from zero, No CUDA experience needed. The book notes that most inference work today happens at the inference-engine level.
How long does Inference Engineering take?
Expect roughly ~256 pages (~46,000 words); ~4-5 hours to read, a weekend with the calculators and exercises. Most learners work through it at their own pace.
What will I learn from Inference Engineering?
You'll learn: Decide between shared model APIs and dedicated deployments using latency budgets, traffic patterns and unit economics; Measure time-to-first-token and tokens-per-second correctly before you start optimizing any deployment; Use arithmetic intensity and the ops:byte ratio to tell compute-bound from memory-bound inference workloads; Compare GPU generations from Hopper to Blackwell and Rubin, plus AMD GPUs, TPUs and multi-GPU interconnects; Place vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo in the serving stack and load-test them realistically; Apply FP8/FP4 quantization, speculative decoding, prefix caching and cache-aware routing while weighing their tradeoffs; Choose tensor, pipeline, expert or data parallelism, and know when prefill-decode disaggregation pays off; Serve embeddings, speech recognition, text-to-speech, vision-language, image and video models, each with its own bottlenecks; Run production inference with containerization, autoscaling, multi-cloud GPU capacity, safe rollouts and resilient client code.
Topics
Sources
This page was written from 6 sources, 2 on domains other than baseten.co.