TRL: Transformer Reinforcement Learning
by Hugging Face
The reference for post-training: SFT, DPO, GRPO and thirteen more trainers.
Overview
TRL is Hugging Face's post-training library, and its documentation is the reference for practitioners who need to take a pretrained model and align it. The docs open with a taxonomy that is the fastest way to orient yourself in the field: online methods (GRPOTrainer, RLOOTrainer, OnlineDPOTrainer, NashMDTrainer, PPOTrainer, XPOTrainer), reward modelling (RewardTrainer, PRMTrainer), offline methods (SFTTrainer, DPOTrainer, KTOTrainer, BCOTrainer, CPOTrainer, ORPOTrainer) and knowledge distillation (GKDTrainer, MiniLLMTrainer), each flagged for vLLM support and experimental status so you can tell a production-ready trainer from a research one at a glance. The site is organised into Getting Started, Conceptual Guides covering dataset formats, a training FAQ and how to read training logs (the section most people skip and most need), How-to Guides on reducing memory, speeding up and distributing training, Integrations with DeepSpeed, Liger Kernel and PEFT, Examples, and a full API reference. TRL sits directly on top of Transformers and Accelerate and composes with PEFT for QLoRA workflows, so the same script scales from a single consumer GPU to a multi-node run. It is maintained by Hugging Face, has passed 19,000 GitHub stars and reached a stable v1 API, and recent releases in the v1.9 line added multi-environment agentic reinforcement learning through Harbor and OpenEnv, streaming dataset support for GRPO and RLOO, and promoted KTO out of experimental status.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- Reference docs; ~2-3 hours for the quickstart plus one trainer guide
- Provider
- Hugging Face
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose between SFT, DPO, GRPO, KTO and RLOO for a given alignment goal
- ✓Format a preference or prompt-completion dataset the way each trainer expects
- ✓Read TRL training logs and diagnose a run that is diverging
- ✓Cut memory use with gradient checkpointing and the PEFT QLoRA integration
- ✓Distribute training across GPUs using Accelerate and DeepSpeed configurations
- ✓Speed up online RL rollouts with co-located vLLM inference servers
- ✓Train a reward model and use it inside an online RL loop
Highlights
- •The trainer taxonomy flags vLLM support and experimental status per method, so production and research code are distinguishable at a glance
- •Covers the current generation of methods including GRPO, RLOO, KTO and ORPO, not just the 2023 PPO-RLHF pipeline
- •Conceptual guides on dataset formats and reading training logs answer the questions that actually break runs
- •First-class integration with PEFT, DeepSpeed, Liger Kernel and vLLM, so it composes with the rest of the Hugging Face stack
- •Maintained in the open at 19k GitHub stars with frequent releases, so the docs track the library rather than lagging it
Who It’s For
Best For
- ✓ML engineers running their first post-training or alignment job
- ✓Researchers comparing DPO, GRPO and PPO on the same base model
- ✓Teams building a domain-specific instruct model on a limited GPU budget
- ✓Transformers users who need to go past supervised fine-tuning
Prerequisites
- •Solid PyTorch and Hugging Face Transformers experience; this is not an entry point to either
- •Working understanding of language model training: tokenisation, loss, batching, learning-rate schedules
- •GPU access, since most trainers assume at least one modern accelerator
FAQ
What is TRL: Transformer Reinforcement Learning?
Hugging Face's official documentation for TRL, the library most teams reach for when a pretrained model needs alignment rather than more pretraining. It documents sixteen trainers spanning supervised fine-tuning, preference optimisation, online reinforcement learning, reward modelling and distillation, plus the dataset formats, memory techniques and distributed setups each one needs. Read it when you have a base model and need it to follow instructions or match a preference signal.
Is TRL: Transformer Reinforcement Learning free?
TRL: Transformer Reinforcement Learning is free to access.
What level is TRL: Transformer Reinforcement Learning for?
TRL: Transformer Reinforcement Learning is aimed at a advanced audience. Recommended background: Solid PyTorch and Hugging Face Transformers experience; this is not an entry point to either, Working understanding of language model training: tokenisation, loss, batching, learning-rate schedules, GPU access, since most trainers assume at least one modern accelerator.
How long does TRL: Transformer Reinforcement Learning take?
Expect roughly Reference docs; ~2-3 hours for the quickstart plus one trainer guide. Most learners work through it at their own pace.
What will I learn from TRL: Transformer Reinforcement Learning?
You'll learn: Choose between SFT, DPO, GRPO, KTO and RLOO for a given alignment goal; Format a preference or prompt-completion dataset the way each trainer expects; Read TRL training logs and diagnose a run that is diverging; Cut memory use with gradient checkpointing and the PEFT QLoRA integration; Distribute training across GPUs using Accelerate and DeepSpeed configurations; Speed up online RL rollouts with co-located vLLM inference servers; Train a reward model and use it inside an online RL loop.
Topics
Sources
This page was written from 3 sources, 2 on domains other than huggingface.co.