Megatron-LM and Megatron Core
by NVIDIA
NVIDIA's reference stack for pretraining transformers across thousands of GPUs — the parallelism, not just the theory.
Overview
Megatron-LM is NVIDIA's reference implementation for training transformer models at scale, and it splits into two layers that matter when you decide how to adopt it. Megatron-LM itself is the reference example — Megatron Core plus pre-configured training scripts such as pretrain_gpt.py, pretrain_mamba.py, pretrain_vlm.py and train_rl.py — aimed at research teams learning distributed training or running quick experiments. Megatron Core is the composable library underneath: GPU-optimized transformer building blocks, five parallelism strategies (tensor, pipeline, data, expert and context parallelism), mixed-precision support spanning FP16, BF16, FP8 and FP4, fused kernels, distributed optimizers, activation offloading and high-performance dataloaders, intended for teams building their own training framework rather than adopting NVIDIA's loop. Supported architectures include GPT-style decoders, Mamba, vision-language models, and mixture-of-experts designs modelled on DeepSeek-V3, Qwen3 and Mixtral, plus hybrids such as Falcon-H1. The published performance envelope is up to 47% Model FLOP Utilization on H100 clusters, with a 462B-parameter model trained across 6,144 H100 GPUs. Documentation lives in the repository and at docs.nvidia.com, organised as About, Get Started (installation and quickstart), Basic Usage (data preparation, training examples, parallelism strategies), Supported Models, Advanced Features (mixture of experts, context parallelism, FSDP, distributed optimizers, activation offloading), a Developer Guide and a full API reference. NGC containers are the supported installation path, and the repo carries 17.5k stars.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~1 day for the single-node quickstart; weeks for a real multi-node pretraining run
- Provider
- NVIDIA
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose between tensor, pipeline, data, expert and context parallelism for a given model and cluster
- ✓Configure and launch a multi-node GPT pretraining run from NVIDIA's reference scripts
- ✓Prepare and shard a pretraining corpus using Megatron's high-performance dataloaders
- ✓Train mixture-of-experts models using expert parallelism and the distributed optimizer
- ✓Apply FP8 and FP4 mixed precision without destabilising a long training run
- ✓Use activation offloading and recomputation to fit larger models into GPU memory
- ✓Embed Megatron Core building blocks inside a training framework of your own
Highlights
- •17.5k stars, and the parallelism reference that other stacks (NeMo, lm-eval backends, HF Accelerate) integrate against
- •Documents a real 462B-parameter run across 6,144 H100 GPUs at up to 47% MFU — measured, not extrapolated
- •Ships working pretrain scripts for GPT, Mamba, VLM and RL, not just library primitives
- •Megatron Core is deliberately separable, so you can adopt the parallelism without adopting NVIDIA's training loop
- •Covers current MoE and hybrid architectures (DeepSeek-V3, Qwen3, Mixtral, Falcon-H1) rather than 2020-era GPT only
Who It’s For
Best For
- ✓Teams pretraining or continued-pretraining a model larger than a single node can hold
- ✓Infrastructure engineers who need to reason about parallelism trade-offs concretely
- ✓Framework builders composing their own trainer from GPU-optimized primitives
- ✓Readers of distributed-training papers who want the reference implementation alongside
Prerequisites
- •Strong PyTorch and distributed-training background — this is not an introduction
- •Access to multi-GPU NVIDIA hardware; H100-class clusters for the documented performance
- •Comfort with containers (NGC) and cluster schedulers such as Slurm
FAQ
What is Megatron-LM and Megatron Core?
The codebase behind much of the industry's large-scale pretraining. Megatron-LM gives you working pretrain scripts and Megatron Core gives you composable, GPU-optimized building blocks — tensor, pipeline, data, expert and context parallelism, FP8 and FP4 mixed precision, fused kernels and high-performance dataloaders — for training dense, mixture-of-experts and hybrid models from a few billion to several hundred billion parameters.
Is Megatron-LM and Megatron Core free?
Megatron-LM and Megatron Core is free to access.
What level is Megatron-LM and Megatron Core for?
Megatron-LM and Megatron Core is aimed at a advanced audience. Recommended background: Strong PyTorch and distributed-training background — this is not an introduction, Access to multi-GPU NVIDIA hardware; H100-class clusters for the documented performance, Comfort with containers (NGC) and cluster schedulers such as Slurm.
How long does Megatron-LM and Megatron Core take?
Expect roughly ~1 day for the single-node quickstart; weeks for a real multi-node pretraining run. Most learners work through it at their own pace.
What will I learn from Megatron-LM and Megatron Core?
You'll learn: Choose between tensor, pipeline, data, expert and context parallelism for a given model and cluster; Configure and launch a multi-node GPT pretraining run from NVIDIA's reference scripts; Prepare and shard a pretraining corpus using Megatron's high-performance dataloaders; Train mixture-of-experts models using expert parallelism and the distributed optimizer; Apply FP8 and FP4 mixed precision without destabilising a long training run; Use activation offloading and recomputation to fit larger models into GPU memory; Embed Megatron Core building blocks inside a training framework of your own.
Topics
Sources
This page was written from 2 sources, 1 on domains other than github.com.