DeepSpeed Documentation and Tutorials
by DeepSpeed (deepspeedai)
The reference for ZeRO, offloading and 3D parallelism — how to train or fine-tune a model that does not fit on your GPUs.
Overview
DeepSpeed is a deep-learning optimization library, originating at Microsoft Research and now developed under the deepspeedai GitHub organisation, organised around four pillars — Training, Inference, Compression and Science. The documentation site pairs conceptual pages with a tutorial library of roughly 47 entries: 7 under Getting Started, 34 under Training, 5 under Inference and 1 under Compression. The training track covers ZeRO, ZeRO++, Mixed Precision ZeRO++ and ZeRO-Offload; 3D parallelism through Pipeline Parallelism, Automatic Tensor Parallelism and the Megatron-LM GPT-2 walkthrough; long-context work via DeepSpeed-Ulysses, Ulysses-Offload, Arctic Long Sequence Training and Sparse Attention; Mixture-of-Experts training, MoE for NLG and MoE inference; communication-efficient optimizers (1-Bit Adam, 1-Bit LAMB, 0/1 Adam, Domino); and operational concerns including Universal Checkpointing, DataStates-LLM checkpointing, DeepNVMe, Autotuning, the Flops Profiler, PyTorch Profiler integration, communication logging and the accelerator abstraction interface for non-NVIDIA hardware. Introductory tutorials start at CIFAR-10 and DCGAN before moving to BERT pre-training and BingBertSQuAD fine-tuning. The central design claim, repeated across the ZeRO tutorials, is that memory optimisation is a configuration change: you enable a ZeRO stage in the DeepSpeed JSON config without modifying model code. The repository has 42.9k stars and 3,307 commits, and the site's news feed runs current through May 2026 with entries on the Muon optimizer, System DMA for ZeRO-3 on AMD GPUs, SuperOffload and the ZenFlow offloading engine.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~1 hour for Getting Started + ZeRO; ~10-15 hours to work through the 40+ tutorials relevant to your setup
- Provider
- DeepSpeed (deepspeedai)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose between ZeRO stages 1, 2 and 3 based on what actually limits your memory
- ✓Offload optimizer state and parameters to CPU and NVMe with ZeRO-Offload and ZeRO-Infinity
- ✓Combine data, tensor and pipeline parallelism into a 3D-parallel training configuration
- ✓Configure all of the above through the DeepSpeed JSON config without rewriting model code
- ✓Train multi-million-token sequences using DeepSpeed-Ulysses and Arctic Long Sequence Training
- ✓Train and serve Mixture-of-Experts models, including MoE inference paths
- ✓Profile a training step with the Flops Profiler, PyTorch Profiler and communication logging
- ✓Save and reload checkpoints across different parallelism layouts with Universal Checkpointing
Highlights
- •The canonical primary source for ZeRO — the memory technique nearly every other training stack reimplements or wraps
- •Tutorials are configuration-first, so most wins land as a JSON change rather than a model rewrite
- •Covers the unglamorous production problems: checkpoint portability, NVMe I/O, autotuning and profiling
- •Actively developed and current — news items through May 2026 including Muon optimizer support and AMD SDMA work
- •Accelerator abstraction pages make the AMD and non-NVIDIA path explicit rather than an afterthought
Who It’s For
Best For
- ✓ML engineers pre-training or full-fine-tuning models too large for a single GPU
- ✓Infrastructure engineers tuning multi-node training throughput and memory
- ✓Researchers working on long-context or Mixture-of-Experts architectures
Prerequisites
- •Solid PyTorch, including writing your own training loop
- •Access to multi-GPU hardware — most tutorials cannot be run meaningfully on one consumer GPU
- •Working understanding of data parallelism, mixed precision and optimizer state
FAQ
What is DeepSpeed Documentation and Tutorials?
The official documentation and tutorial library for DeepSpeed, the distributed training and inference optimization library behind ZeRO, ZeRO-Offload and ZeRO-Infinity. Written for engineers who have hit an out-of-memory wall and need to shard optimizer state, gradients and parameters across GPUs or offload them to CPU and NVMe. After working through it you can configure ZeRO stages via JSON, combine data, tensor and pipeline parallelism, and profile where your training step is actually spending time.
Is DeepSpeed Documentation and Tutorials free?
DeepSpeed Documentation and Tutorials is free to access.
What level is DeepSpeed Documentation and Tutorials for?
DeepSpeed Documentation and Tutorials is aimed at a advanced audience. Recommended background: Solid PyTorch, including writing your own training loop, Access to multi-GPU hardware — most tutorials cannot be run meaningfully on one consumer GPU, Working understanding of data parallelism, mixed precision and optimizer state.
How long does DeepSpeed Documentation and Tutorials take?
Expect roughly ~1 hour for Getting Started + ZeRO; ~10-15 hours to work through the 40+ tutorials relevant to your setup. Most learners work through it at their own pace.
What will I learn from DeepSpeed Documentation and Tutorials?
You'll learn: Choose between ZeRO stages 1, 2 and 3 based on what actually limits your memory; Offload optimizer state and parameters to CPU and NVMe with ZeRO-Offload and ZeRO-Infinity; Combine data, tensor and pipeline parallelism into a 3D-parallel training configuration; Configure all of the above through the DeepSpeed JSON config without rewriting model code; Train multi-million-token sequences using DeepSpeed-Ulysses and Arctic Long Sequence Training; Train and serve Mixture-of-Experts models, including MoE inference paths; Profile a training step with the Flops Profiler, PyTorch Profiler and communication logging; Save and reload checkpoints across different parallelism layouts with Universal Checkpointing.
Topics
Sources
This page was written from 3 sources, 1 on domains other than deepspeed.ai.