DAPO: An Open-Source LLM Reinforcement Learning System at Scale
by ByteDance Seed & Tsinghua AIR (SIA-Lab)
A fully open-sourced, reproducible RL recipe for training reasoning LLMs at scale.
Overview
DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) introduces an RL algorithm and complete open-source system for eliciting complex reasoning in large language models. The paper details four key techniques that make large-scale RL training stable and effective, reporting 50 points on AIME 2024 with a Qwen2.5-32B base model while using roughly half the training steps of prior work. Unlike closed efforts such as OpenAI's o1 and DeepSeek-R1 that withhold training details, the authors fully open-source the recipe: algorithm design, training code built on the verl framework, the curated DAPO-Math-17k dataset, verifiers, and model weights. Submitted to arXiv in March 2025 (revised May 2025), it is a practical reference for reproducing and building on state-of-the-art reasoning-model training rather than depending on proprietary black boxes.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~1 hour read
- Provider
- ByteDance Seed & Tsinghua AIR (SIA-Lab)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓The four core techniques (including decoupled clipping and dynamic sampling) that stabilize large-scale RL for reasoning LLMs
- ✓How to reproduce a state-of-the-art reasoning RL run end to end with open code, data, and verifiers
- ✓Why token-level policy-gradient details and reward shaping matter for long chain-of-thought training
- ✓How DAPO compares to o1- and DeepSeek-R1-style approaches on benchmarks like AIME 2024
Highlights
- •Fully open-sourced: algorithm, verl-based training code, DAPO-Math-17k dataset, verifiers, and model weights
- •Reproducible SOTA — 50 points on AIME 2024 with a Qwen2.5-32B base in far fewer steps
- •One of the most-cited practical references for LLM RL post-training in 2025
Who It’s For
Best For
- ✓ML engineers and researchers doing RL post-training or building reasoning models
- ✓Practitioners who want a reproducible alternative to closed reasoning-RL recipes
Prerequisites
- •Solid grasp of reinforcement learning (policy-gradient, PPO/GRPO-style methods)
- •Experience fine-tuning or post-training LLMs
- •Familiarity with reasoning benchmarks and chain-of-thought
FAQ
What is DAPO: An Open-Source LLM Reinforcement Learning System at Scale?
A landmark 2025 paper from ByteDance Seed and Tsinghua AIR that fully open-sources a large-scale reinforcement learning system for training reasoning LLMs. Essential reading for engineers doing RL post-training who want a reproducible recipe rather than the withheld details behind o1 and DeepSeek-R1.
Is DAPO: An Open-Source LLM Reinforcement Learning System at Scale free?
DAPO: An Open-Source LLM Reinforcement Learning System at Scale is free to access.
What level is DAPO: An Open-Source LLM Reinforcement Learning System at Scale for?
DAPO: An Open-Source LLM Reinforcement Learning System at Scale is aimed at a advanced audience. Recommended background: Solid grasp of reinforcement learning (policy-gradient, PPO/GRPO-style methods), Experience fine-tuning or post-training LLMs, Familiarity with reasoning benchmarks and chain-of-thought.
How long does DAPO: An Open-Source LLM Reinforcement Learning System at Scale take?
Expect roughly ~1 hour read. Most learners work through it at their own pace.
What will I learn from DAPO: An Open-Source LLM Reinforcement Learning System at Scale?
You'll learn: The four core techniques (including decoupled clipping and dynamic sampling) that stabilize large-scale RL for reasoning LLMs; How to reproduce a state-of-the-art reasoning RL run end to end with open code, data, and verifiers; Why token-level policy-gradient details and reward shaping matter for long chain-of-thought training; How DAPO compares to o1- and DeepSeek-R1-style approaches on benchmarks like AIME 2024.