ModelsMLFine-Tuning

Quantization and Fast Inference: A Practitioner's Guide to Efficient AI

by Manning Publications (Vivek Kalyanarangan)

AdvancedBookPaid12 chapters, ~339 pages (estimated), self-paced; per-chapter code in the companion repo

Quantize models from first principles to a 4-bit LLM server and an on-device pipeline, with PTQ, QAT, GPTQ/AWQ, GGUF and vLLM

Start LearningAdded Nov 22, 2026 · Updated Nov 22, 2026

Overview

Quantization and Fast Inference is a Manning book by Vivek Kalyanarangan, an AI/ML architect with more than twelve years of experience building large-scale ML systems. Manning lists it as a complete MEAP (all chapters available, last updated October 2026) with print estimated for December 2026 at about 339 pages. The companion repo lists twelve chapters. It opens with the efficiency problem and builds quantization from first principles, then covers granularity choices, calibration and post-training quantization, quantization-aware training, and quantization pathways across frameworks. Chapter 7 applies all of this to large language models, chapter 8 covers sub-8-bit formats, and the remaining chapters handle deployment pipelines, CPU-friendly LLM serving with llama.cpp and GGUF, and edge and mobile targets. The final chapter delivers a four-bit LLM server and an on-device vision pipeline. Manning's description adds cross-framework parity testing and LLM trade-offs such as activation outliers, KV-cache pressure and NF4/FP4 formats. The MIT-licensed repo (folders ch1 to ch12) uses PyTorch, Transformers and TorchVision with TorchAO, ONNX Runtime and Optimum, TensorFlow Lite, bitsandbytes, gptqmodel, autoawq, TensorRT, OpenVINO, llama.cpp/GGUF, Core ML, MLX, vLLM, and Prometheus and Grafana for monitoring. Worked models range from ResNet-18 and BERT to TinyLlama, OPT, Llama-2-7B, Llama-3.2-1B and an AWQ build of Qwen2.5-3B-Instruct. Manning aims it at engineers with roughly two to six years of experience. The README asks for Python 3.9+ and basic PyTorch and lists a GPU as optional.

At a Glance

Topic
Models
Level
Advanced
Format
Book
Cost
Paid
Duration
12 chapters, ~339 pages (estimated), self-paced; per-chapter code in the companion repo
Provider
Manning Publications (Vivek Kalyanarangan)
Hands-on
Yes — code/exercises
Certificate
None

What You’ll Learn

  • ✓Implement quantization from first principles, including scales, zero points and rounding error
  • ✓Choose quantization granularity per tensor, per channel or per group and measure the accuracy cost
  • ✓Calibrate and apply post-training quantization, and recover accuracy with quantization-aware training
  • ✓Quantize large language models with GPTQ, AWQ and bitsandbytes and handle activation outliers
  • ✓Work with sub-8-bit formats such as NF4 and FP4 and judge their quality trade-offs
  • ✓Move a quantized model across PyTorch, ONNX Runtime, TensorRT and OpenVINO and test output parity
  • ✓Serve quantized LLMs on CPU with llama.cpp and GGUF, and on GPU with vLLM
  • ✓Deploy quantized models to edge and mobile targets with Core ML, MLX and TensorFlow Lite

Highlights

  • •Covers both classic vision and encoder models (ResNet-18, BERT) and LLMs (Llama-3.2-1B, Qwen2.5-3B AWQ), so the methods are shown working outside LLMs too
  • •Spans the toolchain most courses skip: TorchAO, ONNX Runtime, TensorRT, OpenVINO, llama.cpp/GGUF, Core ML, MLX and vLLM
  • •Ends with two complete builds, a four-bit LLM server and an on-device vision pipeline, rather than isolated snippets
  • •Includes cross-framework parity testing and Prometheus/Grafana monitoring for quantized deployments
  • •MIT-licensed repo with one folder per chapter; per the README, a GPU is optional for much of it

Who It’s For

Best For

  • ✓ML engineers whose model is too slow or too large for its serving budget
  • ✓Infrastructure engineers deploying LLMs on CPUs, consumer GPUs or edge devices
  • ✓Applied researchers who want to understand why a 4-bit model lost accuracy and how to recover it
  • ✓Teams choosing between GPTQ, AWQ, bitsandbytes and GGUF for a production model

Prerequisites

  • •Comfortable Python and two or more years of ML engineering experience
  • •Working PyTorch knowledge, including having built and trained neural networks
  • •Optional GPU; the README lists Python 3.9+ and basic PyTorch as the minimum setup

FAQ

What is Quantization and Fast Inference: A Practitioner's Guide to Efficient AI?

A Manning book for ML and infrastructure engineers with a few years of experience who need to make models smaller and faster without redesigning them. It builds quantization from first principles, then walks through post-training quantization, quantization-aware training, LLM-specific formats and deployment on GPUs, CPUs and edge devices, ending with a 4-bit LLM server and an on-device vision pipeline.

Is Quantization and Fast Inference: A Practitioner's Guide to Efficient AI free?

Quantization and Fast Inference: A Practitioner's Guide to Efficient AI is a paid resource.

What level is Quantization and Fast Inference: A Practitioner's Guide to Efficient AI for?

Quantization and Fast Inference: A Practitioner's Guide to Efficient AI is aimed at a advanced audience. Recommended background: Comfortable Python and two or more years of ML engineering experience, Working PyTorch knowledge, including having built and trained neural networks, Optional GPU; the README lists Python 3.9+ and basic PyTorch as the minimum setup.

How long does Quantization and Fast Inference: A Practitioner's Guide to Efficient AI take?

Expect roughly 12 chapters, ~339 pages (estimated), self-paced; per-chapter code in the companion repo. Most learners work through it at their own pace.

What will I learn from Quantization and Fast Inference: A Practitioner's Guide to Efficient AI?

You'll learn: Implement quantization from first principles, including scales, zero points and rounding error; Choose quantization granularity per tensor, per channel or per group and measure the accuracy cost; Calibrate and apply post-training quantization, and recover accuracy with quantization-aware training; Quantize large language models with GPTQ, AWQ and bitsandbytes and handle activation outliers; Work with sub-8-bit formats such as NF4 and FP4 and judge their quality trade-offs; Move a quantized model across PyTorch, ONNX Runtime, TensorRT and OpenVINO and test output parity; Serve quantized LLMs on CPU with llama.cpp and GGUF, and on GPU with vLLM; Deploy quantized models to edge and mobile targets with Core ML, MLX and TensorFlow Lite.

Topics

quantizationLLM inferenceGPTQAWQllama.cppvLLM

Sources

This page was written from 2 sources, 1 on domains other than manning.com.

  1. 1.manning.com — quantization and fast inferencevendor
  2. 2.github.com — Efficient AI in Practice Quantization and Fast Inference