DAPO: An Open-Source LLM Reinforcement Learning System at Scale
by ByteDance Seed, Tsinghua AIR & The University of Hong Kong (SIA-Lab)
The four RL techniques that o1 and R1 write-ups withheld, stated plainly and fully open-sourced.
Overview
DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) is the paper and open-source system from ByteDance Seed, Tsinghua University's Institute for AI Industry Research and the University of Hong Kong that made large-scale reasoning RL reproducible. Its premise is that the key details behind o1-class and R1-class reasoning models were concealed, so the community could not replicate the training results. The paper names four concrete techniques that make it work: Clip-Higher, which decouples the upper and lower PPO clipping bounds to preserve exploration; Dynamic Sampling, which filters out prompts whose rollouts are all-correct or all-wrong and therefore carry no gradient; Token-level Policy Gradient Loss, which weights long chains of thought properly instead of averaging per sequence; and Overlong Reward Shaping, which stops truncated generations from injecting reward noise. Trained on a Qwen2.5-32B base model across 128 H20 GPUs, the system reaches 50 points on AIME 2024, beating the previous DeepSeek-R1-Zero-Qwen-32B state of the art while using 50% fewer training steps. Everything needed to reproduce it is released: the algorithm implementation on top of the verl framework, the curated DAPO-Math-17k dataset, model weights, the rule-based verifier and the training scripts. Submitted March 2025 and revised May 2025; the companion GitHub repository carries roughly 1.9k stars.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~60 min read, plus the released code and dataset if you reproduce it
- Provider
- ByteDance Seed, Tsinghua AIR & The University of Hong Kong (SIA-Lab)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why decoupling the PPO clip bounds (Clip-Higher) preserves exploration during RL
- ✓How dynamic sampling removes zero-gradient prompts and stabilises the training batch
- ✓Why token-level policy gradient loss matters for long chain-of-thought responses
- ✓How overlong reward shaping prevents truncated generations from poisoning the reward signal
- ✓Reproducing a large-scale RL run on the verl framework end to end
- ✓How a rule-based verifier and curated math dataset replace a learned reward model
Highlights
- •The four techniques are stated plainly, which is exactly what the o1 and R1 write-ups withheld
- •Fully open: algorithm, verl-based training code, DAPO-Math-17k, weights, verifier and scripts
- •Matches the prior 32B state of the art on AIME 2024 using 50% fewer training steps
- •Concrete hardware numbers — 128 H20 GPUs on Qwen2.5-32B — so you can size a reproduction
- •Backed by a maintained GitHub repository (~1.9k stars) and a project page with results
Who It’s For
Best For
- ✓ML engineers running reinforcement learning on reasoning models
- ✓Teams reproducing R1-style training without a proprietary recipe
- ✓Researchers who need a documented, working large-scale RL baseline
Prerequisites
- •Working knowledge of PPO or GRPO-style policy-gradient reinforcement learning
- •Experience with multi-node GPU training and a framework such as verl
FAQ
What is DAPO: An Open-Source LLM Reinforcement Learning System at Scale?
The paper and complete open-source system that made large-scale reasoning reinforcement learning reproducible. It names four specific techniques behind a 50-point AIME 2024 result on Qwen2.5-32B, and releases the verl-based training code, the DAPO-Math-17k dataset, model weights and the rule-based verifier. For ML engineers running RL on reasoning models rather than reading about it.
Is DAPO: An Open-Source LLM Reinforcement Learning System at Scale free?
DAPO: An Open-Source LLM Reinforcement Learning System at Scale is free to access.
What level is DAPO: An Open-Source LLM Reinforcement Learning System at Scale for?
DAPO: An Open-Source LLM Reinforcement Learning System at Scale is aimed at a advanced audience. Recommended background: Working knowledge of PPO or GRPO-style policy-gradient reinforcement learning, Experience with multi-node GPU training and a framework such as verl.
How long does DAPO: An Open-Source LLM Reinforcement Learning System at Scale take?
Expect roughly ~60 min read, plus the released code and dataset if you reproduce it. Most learners work through it at their own pace.
What will I learn from DAPO: An Open-Source LLM Reinforcement Learning System at Scale?
You'll learn: Why decoupling the PPO clip bounds (Clip-Higher) preserves exploration during RL; How dynamic sampling removes zero-gradient prompts and stabilises the training batch; Why token-level policy gradient loss matters for long chain-of-thought responses; How overlong reward shaping prevents truncated generations from poisoning the reward signal; Reproducing a large-scale RL run on the verl framework end to end; How a rule-based verifier and curated math dataset replace a learned reward model.
Topics
Sources
This page was written from 3 sources, 2 on domains other than arxiv.org.