prime-rl Documentation: Asynchronous Agentic RL Training at Scale

by Prime Intellect

AdvancedDocumentationFree~4-6 hours to read the docs and run the 1-2 GPU smoke tests; multi-node runs take days of setup

The open framework Prime Intellect used to post-train INTELLECT-3, with docs on async RL, loss design and scaling to large GPU clusters.

Start LearningAdded Nov 14, 2026 · Updated Nov 14, 2026

Overview

prime-rl is Prime Intellect's Apache-2.0 framework for asynchronous reinforcement learning of LLMs, described in its README as agentic RL training at scale and aimed at large MoE models on big GPU clusters. Prime Intellect says INTELLECT-3, a 106B-parameter MoE post-trained with SFT and RL on the GLM 4.5 Air base model and announced November 26, 2025, was trained end to end with prime-rl on 512 H200 GPUs across 64 nodes. The docs folder covers Overview, Configuration, Training, Eval, Inference, Scaling, Algorithms, Advanced and Development. The architecture splits a run into three processes: a vLLM inference server holding the current policy, a CPU orchestrator that runs multi-turn rollouts in verifiers environments, computes advantages and packs batches, and an FSDP2 trainer with expert and context parallelism, FP8 on Hopper and LoRA. Rollouts move over ZMQ and weights over NCCL. The Algorithms page explains per-token staleness bounds for off-policy training (max_staleness defaults to 8), the IPO and IcePop losses, GRPO-style group advantages, hierarchical GRPO, self-play advantages and curricula. The Training page covers the rl and sft entrypoints, checkpointing including async saves and live config updates, and W&B monitoring. Scaling covers Slurm launches, expert and context parallelism and disaggregated inference. You need at least one NVIDIA GPU, and the full RL smoke test needs two. As of November 14, 2026 the repo had about 2.2k stars and 2,868 commits, and its model table lists Qwen3.5 and GLM-5.

At a Glance

Topic: Fine-Tuning
Level: Advanced
Format: Documentation
Cost: Free
Duration: ~4-6 hours to read the docs and run the 1-2 GPU smoke tests; multi-node runs take days of setup
Provider: Prime Intellect
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓How an async RL run splits into inference, orchestrator and trainer processes and how they communicate
✓Launch SFT and RL runs from TOML configs with the uv run sft and uv run rl entrypoints
✓Plug verifiers environments from the Environments Hub into multi-turn agentic RL training
✓Bound off-policy staleness per token and read the mismatch KL metrics that reveal drift
✓Choose between IPO and IcePop losses and configure GRPO, hierarchical GRPO or self-play advantages
✓Enable, resume, pause and export checkpoints, including asynchronous saves during long runs
✓Scale runs across Slurm nodes with expert parallelism and Ulysses-style context parallelism
✓Monitor training with W&B and the local dashboard and spot whether trainer or orchestrator is the bottleneck

Highlights

•Production-tested: Prime Intellect reports INTELLECT-3, a 106B MoE, was trained end to end with it on 512 H200 GPUs
•Algorithm docs go deeper than most RL frameworks, covering staleness, importance-ratio masking and advantage variants with config keys
•Runs from a single consumer GPU (tested on RTX 3090 through B200) up to 2048-GPU Slurm examples
•Integrates natively with the verifiers library and Environments Hub, so environments are reusable across training and evals
•Active development as of November 14, 2026: about 2.2k stars, 2,868 commits, and support for Qwen3.5, GLM-5 and GPT-OSS

Who It’s For

Best For

✓ML engineers post-training reasoning or agentic models with reinforcement learning
✓Research teams comparing asynchronous RL algorithms and loss functions on open models
✓Infrastructure engineers running multi-node post-training on Slurm clusters
✓Builders of verifiers environments who want to train against them, not only evaluate

Prerequisites

•Solid PyTorch and LLM fine-tuning experience, including FSDP or other distributed training basics
•Working knowledge of policy-gradient RL for LLMs such as GRPO and PPO-style importance ratios
•At least one NVIDIA GPU (two for the full RL smoke test) and comfort with uv, Linux and Slurm
•Familiarity with vLLM serving and the verifiers environment library helps

FAQ

What is prime-rl Documentation: Asynchronous Agentic RL Training at Scale?

prime-rl is Prime Intellect's open-source framework for large-scale asynchronous reinforcement learning and SFT of LLMs, and its docs explain how to run it. It is for ML engineers post-training reasoning or agentic models. Afterwards you can run RL with verifiers environments, tune the algorithm and scale across Slurm clusters.

Is prime-rl Documentation: Asynchronous Agentic RL Training at Scale free?

prime-rl Documentation: Asynchronous Agentic RL Training at Scale is free to access.

What level is prime-rl Documentation: Asynchronous Agentic RL Training at Scale for?

prime-rl Documentation: Asynchronous Agentic RL Training at Scale is aimed at a advanced audience. Recommended background: Solid PyTorch and LLM fine-tuning experience, including FSDP or other distributed training basics, Working knowledge of policy-gradient RL for LLMs such as GRPO and PPO-style importance ratios, At least one NVIDIA GPU (two for the full RL smoke test) and comfort with uv, Linux and Slurm, Familiarity with vLLM serving and the verifiers environment library helps.

How long does prime-rl Documentation: Asynchronous Agentic RL Training at Scale take?

Expect roughly ~4-6 hours to read the docs and run the 1-2 GPU smoke tests; multi-node runs take days of setup. Most learners work through it at their own pace.

What will I learn from prime-rl Documentation: Asynchronous Agentic RL Training at Scale?

You'll learn: How an async RL run splits into inference, orchestrator and trainer processes and how they communicate; Launch SFT and RL runs from TOML configs with the uv run sft and uv run rl entrypoints; Plug verifiers environments from the Environments Hub into multi-turn agentic RL training; Bound off-policy staleness per token and read the mismatch KL metrics that reveal drift; Choose between IPO and IcePop losses and configure GRPO, hierarchical GRPO or self-play advantages; Enable, resume, pause and export checkpoints, including asynchronous saves during long runs; Scale runs across Slurm nodes with expert parallelism and Ulysses-style context parallelism; Monitor training with W&B and the local dashboard and spot whether trainer or orchestrator is the bottleneck.

Topics

reinforcement learninggrpopost-trainingdistributed trainingvllmagentic rl

Sources

This page was written from 6 sources, 1 on domains other than github.com.