Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning
by Microsoft Research
Bolt reinforcement learning onto the agent you already shipped, without rewriting it for a trainer.
Overview
Agent Lightning is Microsoft Research's MIT-licensed framework for training an agent you have already built, using reinforcement learning, without rewriting it for a trainer. Its premise, set out in the arXiv paper 'Agent Lightning: Train ANY AI Agents with Reinforcement Learning' (Luo et al., submitted August 2025), is a complete decoupling of agent execution from training: agent execution is formulated as a Markov decision process, and LightningRL — a hierarchical RL algorithm with a credit-assignment module — decomposes trajectories generated by any agent into training transitions. The paper claims seamless integration with agents built on LangChain, the OpenAI Agents SDK, AutoGen or from scratch with almost zero code changes, and reports stable gains on text-to-SQL, retrieval-augmented generation and math tool-use tasks. Version 1.0 is a complete redesign and reimplementation down to roughly 3,500 lines of core Python, organised around three components: a Trainer with verl integration for rollout collection, an API Gateway that proxies model traffic so the agent keeps its own tools, context and environment, and a Rollout Controller with local and Kubernetes runners. The documentation runs Installation (base environment and tested GPU stack), a local end-to-end Quick Start, a Basics page on core components, rollouts, events and trajectories, then configuration guides for the trainer, gateway and controller plus asynchronous training with collection and pause/drain behaviour. Worked examples cover Calc-X, GSM8K, ScienceWorld, Search-R1, an LLM-in-sandbox task, and a coding agent taking SWE-bench Verified from 41.8% to 56.4% on 6,000 training samples.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced; ~3-5 hours for the docs and local quick start, plus GPU time for a real training run
- Provider
- Microsoft Research
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Attach RL training to an existing agent without rewriting its control flow
- ✓Understand how LightningRL assigns credit across multi-step and multi-agent trajectories
- ✓Configure the Trainer, API Gateway and Rollout Controller as three separable services
- ✓Collect rollouts, events and trajectories through a model-traffic proxy
- ✓Run rollout workers locally or as Kubernetes Jobs for distributed training
- ✓Set up asynchronous training and reason about collection, pause and drain behaviour
- ✓Reproduce worked examples on GSM8K, ScienceWorld, Search-R1 and a coding agent
Highlights
- •Framework-agnostic by construction: the paper reports integration with LangChain, the OpenAI Agents SDK, AutoGen and hand-rolled agents with almost zero code modification
- •v1.0 is a deliberate shrink to roughly 3,500 lines of core Python after a full rewrite, which makes reading the source alongside the docs realistic
- •Ships a real result rather than a toy: the coding-agent example moves SWE-bench Verified from 41.8% to 56.4% on 6K samples and documents its data cleaning and reward-hacking prevention
- •Native Kubernetes runners, so scaling rollout collection is a deployment concern rather than a rewrite
- •17.9k GitHub stars and MIT-licensed, with the pre-1.0 implementation preserved on a separate branch for anyone mid-migration
Who It’s For
Best For
- ✓ML engineers with a working agent who want it to improve from its own traces
- ✓Teams standardising RL post-training across several different agent frameworks
- ✓Researchers reproducing agentic RL results on SWE-bench, GSM8K or Search-R1
Prerequisites
- •Python and familiarity with at least one agent framework
- •RL post-training fundamentals: rollouts, rewards and policy optimisation
- •GPU access and comfort standing up a verl and vLLM training stack
FAQ
What is Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning?
The official v1.0 documentation for Microsoft Research's Agent Lightning, an MIT-licensed framework that trains an existing agent with reinforcement learning by proxying its model traffic rather than requiring a rewrite. It covers the Trainer, API Gateway and Rollout Controller, asynchronous training, Kubernetes runners, and worked examples up to a coding agent measured on SWE-bench Verified.
Is Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning free?
Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning is free to access.
What level is Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning for?
Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning is aimed at a advanced audience. Recommended background: Python and familiarity with at least one agent framework, RL post-training fundamentals: rollouts, rewards and policy optimisation, GPU access and comfort standing up a verl and vLLM training stack.
How long does Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning take?
Expect roughly Self-paced; ~3-5 hours for the docs and local quick start, plus GPU time for a real training run. Most learners work through it at their own pace.
What will I learn from Agent Lightning Documentation — Train Any AI Agent with Reinforcement Learning?
You'll learn: Attach RL training to an existing agent without rewriting its control flow; Understand how LightningRL assigns credit across multi-step and multi-agent trajectories; Configure the Trainer, API Gateway and Rollout Controller as three separable services; Collect rollouts, events and trajectories through a model-traffic proxy; Run rollout workers locally or as Kubernetes Jobs for distributed training; Set up asynchronous training and reason about collection, pause and drain behaviour; Reproduce worked examples on GSM8K, ScienceWorld, Search-R1 and a coding agent.
Topics
Sources
This page was written from 3 sources, 2 on domains other than microsoft.github.io.