TorchAO Documentation: PyTorch-Native Quantization for Training and Inference
by PyTorch (Meta)
The official guide to quantizing PyTorch models with TorchAO: float8 and MXFP8 training, QAT and int4/FP8/NVFP4 inference.
Overview
TorchAO is PyTorch's native library for quantizing and sparsifying weights, gradients, optimizers and activations, and its documentation at docs.pytorch.org/ao was rebuilt in 2026 (the workflow pages are dated March 25, 2026). The material is organised around three workflows. The training guide covers float8 training of nn.Linear layers with tensorwise, rowwise and rowwise_with_gw_hp recipes, composition with FSDP2, tensor parallelism, torch.compile and activation checkpointing, prototype MXFP8 training, and torchtitan benchmarks on 8x H100, MI300x and B200. The QAT guide explains fake quantization and the prepare/convert flow, QATConfig and custom IntxFakeQuantizeConfig settings, and integrations with Axolotl, Unsloth (including LoRA plus QAT) and the legacy torchtune recipes, with before-and-after scores on Gemma 3, Qwen3 and Llama 3.2. The inference guide lists float8, int8, int4, intx, MXFP8, MXFP4 and NVFP4 configs, accuracy and vLLM throughput benchmarks for Llama-3.1-8B on B200 and H100, ARM CPU kernels, codebook quantization and prototype FP8 attention. A status matrix shows which dtype is stable or prototype on NVIDIA, AMD, Intel and edge hardware. The library is maintained in pytorch/ao on GitHub (about 3,000 stars as of October 13, 2026), with v0.18.0 released on August 3, 2026 adding prototype NVFP4 training.
At a Glance
- Topic
- ML
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced reference: three workflow guides (training, QAT, inference) plus the API reference
- Provider
- PyTorch (Meta)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Apply int4, int8, float8 or NVFP4 quantization to a PyTorch model with the quantize_ API
- ✓Choose between tensorwise, rowwise and rowwise_with_gw_hp float8 training recipes for accuracy versus speed
- ✓Combine float8 training with FSDP2, tensor parallelism, torch.compile and activation checkpointing in torchtitan
- ✓Run quantization-aware training with the prepare and convert flow to recover accuracy lost to int4
- ✓Fine-tune with QAT through Unsloth (full or LoRA) and Axolotl, then export the quantized model
- ✓Read the hardware status matrix to see which dtypes are stable on H100, B200, MI350x and Intel GPUs
- ✓Weigh accuracy, checkpoint size and vLLM prefill and decode speedups for each inference format
- ✓Try prototype MXFP8 training, MXFP8 MoE layers and low-precision FP8 attention on Hopper and Blackwell
Highlights
- •Official PyTorch library already wired into vLLM, SGLang, Hugging Face Transformers, PEFT, diffusers, ExecuTorch, Unsloth and Axolotl
- •Publishes measured numbers with their setup, such as Llama-3.1-8B wikitext perplexity and checkpoint size per format
- •Covers the newest low-precision formats (MXFP8, MXFP4, NVFP4) with an honest stable versus prototype label
- •One library spans pretraining, fine-tuning and serving, so the same configs carry from training to deployment
- •Actively developed: three releases in 2026 (v0.16.0 in February, v0.17.0 in March, v0.18.0 in August)
Who It’s For
Best For
- ✓ML engineers cutting LLM serving cost or memory with int4, float8 or NVFP4 weights
- ✓Teams fine-tuning open models with Unsloth or Axolotl who want quantization-aware training
- ✓Pretraining engineers adding float8 or MXFP8 to large FSDP2 or torchtitan runs
- ✓Inference engineers comparing quantization formats on H100, B200 or AMD hardware
Prerequisites
- •Comfortable PyTorch, including nn.Module, training loops and torch.compile
- •Basic understanding of numeric precision (bf16, float8, int4) and why quantization trades accuracy for speed
- •Access to a recent GPU for most workflows (H100 or B200 for float8 and FP4 formats)
FAQ
What is TorchAO Documentation: PyTorch-Native Quantization for Training and Inference?
TorchAO's documentation is the official guide to PyTorch's native library for low-precision training and inference. It is for ML engineers who fine-tune, pretrain or serve LLMs and want smaller, faster models, and it shows how to apply float8 and MXFP8 training, quantization-aware training with Unsloth or Axolotl, and int4, float8 or NVFP4 inference quantization.
Is TorchAO Documentation: PyTorch-Native Quantization for Training and Inference free?
TorchAO Documentation: PyTorch-Native Quantization for Training and Inference is free to access.
What level is TorchAO Documentation: PyTorch-Native Quantization for Training and Inference for?
TorchAO Documentation: PyTorch-Native Quantization for Training and Inference is aimed at a intermediate audience. Recommended background: Comfortable PyTorch, including nn.Module, training loops and torch.compile, Basic understanding of numeric precision (bf16, float8, int4) and why quantization trades accuracy for speed, Access to a recent GPU for most workflows (H100 or B200 for float8 and FP4 formats).
How long does TorchAO Documentation: PyTorch-Native Quantization for Training and Inference take?
Expect roughly Self-paced reference: three workflow guides (training, QAT, inference) plus the API reference. Most learners work through it at their own pace.
What will I learn from TorchAO Documentation: PyTorch-Native Quantization for Training and Inference?
You'll learn: Apply int4, int8, float8 or NVFP4 quantization to a PyTorch model with the quantize_ API; Choose between tensorwise, rowwise and rowwise_with_gw_hp float8 training recipes for accuracy versus speed; Combine float8 training with FSDP2, tensor parallelism, torch.compile and activation checkpointing in torchtitan; Run quantization-aware training with the prepare and convert flow to recover accuracy lost to int4; Fine-tune with QAT through Unsloth (full or LoRA) and Axolotl, then export the quantized model; Read the hardware status matrix to see which dtypes are stable on H100, B200, MI350x and Intel GPUs; Weigh accuracy, checkpoint size and vLLM prefill and decode speedups for each inference format; Try prototype MXFP8 training, MXFP8 MoE layers and low-precision FP8 attention on Hopper and Blackwell.
Topics
Sources
This page was written from 7 sources, 2 on domains other than docs.pytorch.org.
- 1.docs.pytorch.org — indexvendor
- 2.docs.pytorch.org — indexvendor
- 3.docs.pytorch.org — trainingvendor
- 4.docs.pytorch.org — qatvendor
- 5.docs.pytorch.org — inferencevendor
- 6.github.com — ao
- 7.github.com — releases