SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents

by NovaSky-AI (UC Berkeley Sky Computing Lab, with Anyscale)

AdvancedDocumentationFreeSelf-paced; quickstart is one GRPO run on GSM8K using 4 GPUs, and working through the tutorials plus one recipe takes a few days (estimate)

Train multi-turn, tool-using LLM agents with RL on your own GPUs, including running Tinker scripts unchanged.

Start LearningAdded Sep 26, 2026 · Updated Sep 26, 2026

Overview

SkyRL is an Apache-2.0 reinforcement learning library for training LLMs and multi-turn, tool-using agents, built at UC Berkeley's Sky Computing Lab with Anyscale; its GitHub repo lists skyrl-train (the training framework), skyrl-gym (tool-use environments for math, coding, search and SQL), skyrl-agent (a layer for long-horizon agent training, described in arXiv 2511.16108), skyrl-tx (a cross-platform backend) and a unified skyrl package implementing the Tinker API. The documentation at docs.skyrl.ai is organised into Getting Started (installation, quickstart, system overview, supported models, inference architecture), Tinker integration (overview, quickstart, architecture, cookbook), Tutorials (building a new environment, one-step-off async training, a tools guide), Examples (Megatron, multi-turn text-to-SQL, search), Algorithms (including DAPO), end-to-end Recipes and an API reference. The system overview splits RL into Trainer, Generator, InferenceEngine, Environment and Controller components, with FSDP and Megatron training backends. The quickstart trains Qwen2.5-1.5B-Instruct with GRPO on GSM8K, logging to Weights & Biases, on four GPUs by default. The recipes page publishes reproducible results with commits and W&B links: GSM8K (~0.79 eval accuracy on 4xH100), DAPO on AIME24 across Qwen2.5/3 models up to 32B, SkyRL-SQL on Spider with Qwen2.5-Coder-7B, and SearchR1 question answering. The Tinker integration runs official tinker-cookbook recipes on your own hardware with LoRA or full-parameter training. The project shipped v0.3.0 in July 2026 and added a long-horizon knowledge-work recipe in September 2026.

At a Glance

Topic: Fine-Tuning
Level: Advanced
Format: Documentation
Cost: Free
Duration: Self-paced; quickstart is one GRPO run on GSM8K using 4 GPUs, and working through the tutorials plus one recipe takes a few days (estimate)
Provider: NovaSky-AI (UC Berkeley Sky Computing Lab, with Anyscale)
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓How SkyRL splits RL training into Trainer, Generator, InferenceEngine, Environment and Controller components
✓How to launch a GRPO run on GSM8K with Qwen2.5-1.5B-Instruct and track it in Weights & Biases
✓How to build a custom environment by subclassing BaseTextEnv and implementing step() with rewards
✓How to extend an environment to multi-turn episodes with turn limits, feedback observations and graded rewards
✓How to register environments with skyrl_gym so the training stack can instantiate them by ID
✓How to run official tinker-cookbook scripts unchanged on your own GPUs through SkyRL's Tinker API implementation
✓How to reproduce the DAPO, SkyRL-SQL and SearchR1 recipes against their published benchmark numbers
✓How to scale training with the Megatron backend and one-step-off asynchronous training

Highlights

•Recipes publish reproducible commits and W&B runs with concrete numbers, so you can check that your setup matches theirs
•Research-grade provenance: the SkyRL-Agent paper reports SA-SWE-32B moving from 24.4% to 39.4% Pass@1 on SWE-Bench Verified
•Actively maintained: about 2.3k GitHub stars, 439 forks and 1,300+ commits, with v0.3.0 released July 2026 (checked 2026-09-26)
•Implements the Tinker API, so the same training scripts run on a hosted service or on your own cluster
•Environments are simple text-in/text-out classes, which makes it quicker to plug in your own agentic task than with many RL stacks

Who It’s For

Best For

✓ML engineers post-training open-weight models with GRPO, DAPO or similar RL algorithms
✓Teams RL-training tool-using agents for SQL, search, coding or terminal tasks
✓Researchers who need reproducible baselines and a modular stack they can change
✓Tinker users who want to move the same scripts onto in-house GPUs

Prerequisites

•Solid PyTorch and Python, plus working familiarity with LLM fine-tuning and RL concepts such as policy gradients and GRPO
•Access to multi-GPU hardware: the quickstart defaults to 4 GPUs and the recipes use H100 nodes
•Comfort with distributed tooling such as Ray, uv environments and FSDP or Megatron

FAQ

What is SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents?

The SkyRL documentation covers Berkeley Sky Computing Lab's open-source, full-stack reinforcement learning library for LLMs, written for ML engineers who post-train models and agents. It takes you from a GRPO quickstart to custom multi-turn environments, async training and reproducible recipes, so you can RL-train your own tool-using agent.

Is SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents free?

SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents is free to access.

What level is SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents for?

SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents is aimed at a advanced audience. Recommended background: Solid PyTorch and Python, plus working familiarity with LLM fine-tuning and RL concepts such as policy gradients and GRPO, Access to multi-GPU hardware: the quickstart defaults to 4 GPUs and the recipes use H100 nodes, Comfort with distributed tooling such as Ray, uv environments and FSDP or Megatron.

How long does SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents take?

Expect roughly Self-paced; quickstart is one GRPO run on GSM8K using 4 GPUs, and working through the tutorials plus one recipe takes a few days (estimate). Most learners work through it at their own pace.

What will I learn from SkyRL Documentation — Modular Reinforcement Learning for LLMs and Agents?

You'll learn: How SkyRL splits RL training into Trainer, Generator, InferenceEngine, Environment and Controller components; How to launch a GRPO run on GSM8K with Qwen2.5-1.5B-Instruct and track it in Weights & Biases; How to build a custom environment by subclassing BaseTextEnv and implementing step() with rewards; How to extend an environment to multi-turn episodes with turn limits, feedback observations and graded rewards; How to register environments with skyrl_gym so the training stack can instantiate them by ID; How to run official tinker-cookbook scripts unchanged on your own GPUs through SkyRL's Tinker API implementation; How to reproduce the DAPO, SkyRL-SQL and SearchR1 recipes against their published benchmark numbers; How to scale training with the Megatron backend and one-step-off asynchronous training.

Topics

reinforcement learninggrpoagent trainingpost-trainingtinkerskyrl

Sources

This page was written from 8 sources, 2 on domains other than docs.skyrl.ai.