torchtitan — PyTorch Native Platform for Large-Scale Training
by PyTorch
Read the reference implementation of 4D-parallel LLM pretraining, written in plain PyTorch you can actually follow.
Overview
torchtitan is a PyTorch-native platform for large-scale generative AI training, released under the BSD 3-Clause license with v0.2.0 shipped in October 2025 and an accompanying paper accepted to ICLR 2025. It composes multi-dimensional parallelism — FSDP2 with per-parameter sharding, Tensor Parallel (including async TP), Pipeline Parallel and Context Parallel — so you can apply 4D parallelism to a model with minimal changes to the model code itself. Beyond parallelism it wires in the pieces that make a training run survive contact with a real cluster: distributed checkpointing with async support, torch.compile integration, per-op selective and full activation checkpointing, meta-device initialization, and Float8 plus MXFP8 training for dense and MoE models. Llama 3.1 (8B, 70B, 405B) is supported out of the box, and the published results show speedups over optimized baselines verified at up to 512 GPUs — roughly 65% on Llama 3.1 8B at 128 GPUs (1D), 12.6% on 70B at 256 GPUs (2D), and 30% on 405B at 512 GPUs (3D) on NVIDIA H100s. Its explicit design goal is to be easy to read and extend: modular components instead of a monolithic trainer, which makes it as valuable as a study text on distributed training as it is as a training harness.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~4-8 hours to read the docs and run a first multi-GPU training job
- Provider
- PyTorch
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Compose FSDP2, tensor, pipeline and context parallelism into 4D-parallel training without rewriting model code
- ✓Configure Float8 and MXFP8 training for dense and Mixture-of-Experts models
- ✓Set up distributed and async checkpointing so long runs are resumable after node failures
- ✓Apply torch.compile and per-op selective activation checkpointing to trade compute against memory
- ✓Launch and scale a Llama 3.1-class pretraining run from a single node up to hundreds of GPUs
- ✓Read a production-grade distributed training codebase to understand how the pieces actually fit together
Highlights
- •PyTorch-native and maintained by the PyTorch team — no third-party training framework to adopt
- •ICLR 2025 paper with performance verified at up to 512 GPUs
- •Deliberately modular and readable, so it doubles as a distributed-training reference
- •Float8 / MXFP8 and MoE support that most open training stacks still lack
Who It’s For
Best For
- ✓ML engineers pretraining or continue-training LLMs on multi-node GPU clusters
- ✓Infrastructure engineers evaluating distributed training stacks
- ✓Researchers who need to modify parallelism strategy rather than just call a trainer
- ✓Engineers moving from single-node fine-tuning to large-scale training
Prerequisites
- •Strong PyTorch experience
- •Familiarity with distributed training concepts — data/tensor/pipeline parallelism, sharding
- •Access to multi-GPU hardware to run the examples meaningfully
FAQ
What is torchtitan — PyTorch Native Platform for Large-Scale Training?
The official repository and documentation for torchtitan, PyTorch's native platform for rapid experimentation and large-scale training of generative AI models. It is for ML engineers who need to pretrain or continue-train LLMs across many GPUs and want a readable reference for FSDP2, tensor, pipeline and context parallelism.
Is torchtitan — PyTorch Native Platform for Large-Scale Training free?
torchtitan — PyTorch Native Platform for Large-Scale Training is free to access.
What level is torchtitan — PyTorch Native Platform for Large-Scale Training for?
torchtitan — PyTorch Native Platform for Large-Scale Training is aimed at a advanced audience. Recommended background: Strong PyTorch experience, Familiarity with distributed training concepts — data/tensor/pipeline parallelism, sharding, Access to multi-GPU hardware to run the examples meaningfully.
How long does torchtitan — PyTorch Native Platform for Large-Scale Training take?
Expect roughly ~4-8 hours to read the docs and run a first multi-GPU training job. Most learners work through it at their own pace.
What will I learn from torchtitan — PyTorch Native Platform for Large-Scale Training?
You'll learn: Compose FSDP2, tensor, pipeline and context parallelism into 4D-parallel training without rewriting model code; Configure Float8 and MXFP8 training for dense and Mixture-of-Experts models; Set up distributed and async checkpointing so long runs are resumable after node failures; Apply torch.compile and per-op selective activation checkpointing to trade compute against memory; Launch and scale a Llama 3.1-class pretraining run from a single node up to hundreds of GPUs; Read a production-grade distributed training codebase to understand how the pieces actually fit together.