ART Documentation — Agent Reinforcement Trainer
by OpenPipe
Stop patching prompts: train your agent with GRPO and let an LLM judge write the rewards.
Overview
ART (Agent Reinforcement Trainer) is OpenPipe's open-source framework for improving multi-step LLM agents with reinforcement learning, specifically GRPO, and its documentation is structured as Getting Started (Quick Start, Installation and Setup, Notebooks, FAQ), Fundamentals (Training Loop, Backend, Client, SFT Training, RULER), Features, Experimental, Integrations, Tutorials and Resources. The architecture the Fundamentals section explains is a client/server split: your agent keeps running as ordinary Python against an OpenAI-compatible client on your own machine, while a GPU-side backend handles vLLM inference and Unsloth or torchtune training, so you rarely have to restructure application code to add RL. The loop is inference, trajectory collection, reward assignment, GRPO update, LoRA checkpoint. RULER is the piece worth reading even if you never train a model: it uses an LLM judge to score groups of trajectories relative to one another, removing the hand-written reward function that kills most agent RL projects before they start. Features documents checkpoint forking, automatic deletion of low-performing checkpoints, additional histories for complex rollouts, metric and cost tracking, and MCP-RL for training a model to use a given MCP server; an experimental section covers the GSPO algorithm. Integrations cover LangGraph, OpenEnv, Weights & Biases, Langfuse and OpenPipe. Tutorials train a deep-research agent and a summarizer, and OpenPipe reports its ART-E email agent beating OpenAI o3 — vendor benchmarks, so read them as directional. Apache-2.0, ~10.6k GitHub stars, installed with pip install openpipe-art.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~2 hours to read the fundamentals; a full training notebook is roughly a GPU-day
- Provider
- OpenPipe
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Set up a GRPO training loop around an agent you have already written
- ✓Separate agent rollout code from GPU-side inference and training backends
- ✓Use RULER to generate rewards with an LLM judge instead of hand-written reward functions
- ✓Collect trajectories and assign rewards for multi-step, tool-using agents
- ✓Fork, track and automatically prune LoRA checkpoints across training runs
- ✓Train a model to use a specific MCP server's tools with MCP-RL
- ✓Run supervised fine-tuning as a warm start before reinforcement learning
- ✓Log training metrics and token cost to Weights & Biases or Langfuse
Highlights
- •RULER removes the hardest part of agent RL — writing a reward function — by scoring groups of trajectories with an LLM judge
- •The client/server split lets you keep your agent code as-is and run training on rented GPUs, so experimenting does not mean a rewrite
- •Runnable notebooks for real tasks (email search, deep research, summarization, 2048, Codenames) rather than toy gridworlds
- •Documents what it does not support — Gemma 3 is explicitly excluded — which is rarer in framework docs than it should be
- •Be sceptical of the headline numbers: the ART-E and summarizer benchmark claims are OpenPipe's own, not independently replicated
Who It’s For
Best For
- ✓Engineers whose agent is accurate in demos and unreliable in production
- ✓Teams who want a small open model to match a frontier model on one narrow task
- ✓ML engineers moving from RLHF theory to actually training a tool-using agent
Prerequisites
- •Strong Python and an existing agent built against an OpenAI-compatible API
- •Access to a GPU — rented H100 or A100-class is fine, but this will not run on a laptop alone
- •Working understanding of reinforcement learning basics: policies, rewards, and GRPO or PPO
FAQ
What is ART Documentation — Agent Reinforcement Trainer?
ART's documentation covers OpenPipe's open-source framework for training multi-step LLM agents with reinforcement learning using GRPO. It is written for engineers who have an agent that works sometimes and want the model itself to get better at the task rather than adding another prompt patch. After working through it you can wrap an existing agent in an RL training loop, generate rewards automatically with RULER instead of hand-writing them, and train a small open model on your own trajectories.
Is ART Documentation — Agent Reinforcement Trainer free?
ART Documentation — Agent Reinforcement Trainer is free to access.
What level is ART Documentation — Agent Reinforcement Trainer for?
ART Documentation — Agent Reinforcement Trainer is aimed at a advanced audience. Recommended background: Strong Python and an existing agent built against an OpenAI-compatible API, Access to a GPU — rented H100 or A100-class is fine, but this will not run on a laptop alone, Working understanding of reinforcement learning basics: policies, rewards, and GRPO or PPO.
How long does ART Documentation — Agent Reinforcement Trainer take?
Expect roughly ~2 hours to read the fundamentals; a full training notebook is roughly a GPU-day. Most learners work through it at their own pace.
What will I learn from ART Documentation — Agent Reinforcement Trainer?
You'll learn: Set up a GRPO training loop around an agent you have already written; Separate agent rollout code from GPU-side inference and training backends; Use RULER to generate rewards with an LLM judge instead of hand-written reward functions; Collect trajectories and assign rewards for multi-step, tool-using agents; Fork, track and automatically prune LoRA checkpoints across training runs; Train a model to use a specific MCP server's tools with MCP-RL; Run supervised fine-tuning as a warm start before reinforcement learning; Log training metrics and token cost to Weights & Biases or Langfuse.
Topics
Sources
This page was written from 3 sources, 1 on domains other than art.openpipe.ai.