NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs
by NVIDIA
The post-training stack behind Nemotron — one codebase from a single GPU to thousand-GPU GRPO runs.
Overview
NeMo RL is NVIDIA's Apache-2.0 post-training library for language and vision-language models, and its documentation is structured as About, Environment Setup, End-to-End Examples, Guides, Containers, Development, Design Docs and API Reference. The About section covers overview, performance, model support, features and roadmap, training and generation backends, quick start, installation, the algorithm list, evaluation, cluster setup and a tips-and-tricks page. The algorithm coverage is the reason to read it: GRPO, DAPO, CISPO, PPO, GSPO, GDPO, SFT including LoRA, DPO, reward modelling, on-policy distillation (MOPD) and cross-tokenizer off-policy distillation each get a walkthrough. Two training backends are documented — DTensor, the PyTorch-native path with FSDP2 plus tensor, sequence, pipeline and context parallelism, and Megatron Core with 6D parallelism for models past 100B parameters — and three generation backends: vLLM, SGLang, and native Megatron inference that skips weight conversion. Ray handles distribution, and Environment Setup covers a local workstation as well as Slurm and Kubernetes clusters, so the same configuration scales from one GPU to thousands. More than 25 guides cover Nemotron 3 variants, DeepSeek-V3, ProRLv2, SWE RL, audio post-training, Eagle3 speculative decoding, YaRN long-context training and quantization-aware RL. Release v0.6.0 landed on 30 April 2026 with the SGLang backend, the Muon optimizer and speculative decoding, and NVIDIA trained Nemotron-3-Ultra with the library.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~4-6 hours for the quickstart and one algorithm guide; multi-day for a real cluster run
- Provider
- NVIDIA
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Run GRPO end to end on a math dataset with the provided script, locally or via Slurm
- ✓Choose between the DTensor and Megatron Core backends as model size crosses 100B parameters
- ✓Configure vLLM, SGLang or native Megatron generation and reason about weight-refit cost
- ✓Compare GRPO, DAPO, CISPO, PPO, GSPO and GDPO objectives and when each complexity is justified
- ✓Apply LoRA post-training across backends instead of full-parameter updates on huge models
- ✓Stand up Ray-based clusters on Slurm or Kubernetes and diagnose throughput bottlenecks
- ✓Use on-policy and cross-tokenizer distillation to compress a trained policy into a smaller model
- ✓Evaluate a post-trained checkpoint and reproduce a published recipe such as DeepScaleR
Highlights
- •One codebase spans a single workstation GPU to thousand-GPU clusters — the configuration scales instead of being rewritten
- •The Megatron Core backend brings 6D parallelism, which is what makes 100B+ and MoE post-training tractable at all
- •NVIDIA trained Nemotron-3-Ultra with it, so these are recipes behind a shipped frontier model rather than toy examples
- •More than 25 task guides cover unusual ground: SWE RL, audio post-training, quantization-aware RL, YaRN long-context, speculative decoding
- •Apache-2.0 and shipping fast — v0.6.0 (April 2026) added the SGLang backend, the Muon optimizer and day-0 support for new open models
Who It’s For
Best For
- ✓ML engineers running RLHF or RLVR post-training on multi-GPU clusters
- ✓Teams that outgrew single-node fine-tuning libraries and hit memory or throughput walls
- ✓Researchers reproducing published RL recipes on open-weight models
- ✓Infrastructure engineers standing up Slurm or Kubernetes clusters for post-training workloads
Prerequisites
- •Solid PyTorch and distributed-training experience — FSDP, tensor and pipeline parallelism
- •Working knowledge of RL post-training: policy gradients, reward models, PPO and GRPO
- •Access to multi-GPU NVIDIA hardware; the interesting configurations assume a cluster, not a laptop
FAQ
What is NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs?
NeMo RL is NVIDIA's open-source post-training library, and its documentation is a working reference for reinforcement learning on models from a single GPU up to thousand-GPU clusters. It targets ML engineers and researchers doing RLHF, RLVR and distillation past the point where single-node fine-tuning tools run out of memory. After the quickstart and one algorithm guide you can launch a GRPO run on Slurm or Kubernetes, choose between the DTensor and Megatron backends, pick a generation engine, and evaluate the resulting checkpoint.
Is NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs free?
NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs is free to access.
What level is NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs for?
NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs is aimed at a advanced audience. Recommended background: Solid PyTorch and distributed-training experience — FSDP, tensor and pipeline parallelism, Working knowledge of RL post-training: policy gradients, reward models, PPO and GRPO, Access to multi-GPU NVIDIA hardware; the interesting configurations assume a cluster, not a laptop.
How long does NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs take?
Expect roughly ~4-6 hours for the quickstart and one algorithm guide; multi-day for a real cluster run. Most learners work through it at their own pace.
What will I learn from NeMo RL Documentation — Scalable Post-Training and Reinforcement Learning for LLMs?
You'll learn: Run GRPO end to end on a math dataset with the provided script, locally or via Slurm; Choose between the DTensor and Megatron Core backends as model size crosses 100B parameters; Configure vLLM, SGLang or native Megatron generation and reason about weight-refit cost; Compare GRPO, DAPO, CISPO, PPO, GSPO and GDPO objectives and when each complexity is justified; Apply LoRA post-training across backends instead of full-parameter updates on huge models; Stand up Ray-based clusters on Slurm or Kubernetes and diagnose throughput bottlenecks; Use on-policy and cross-tokenizer distillation to compress a trained policy into a smaller model; Evaluate a post-trained checkpoint and reproduce a published recipe such as DeepScaleR.
Topics
Sources
This page was written from 2 sources, 1 on domains other than docs.nvidia.com.
- 1.docs.nvidia.com — indexvendor
- 2.github.com — RL