RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training
by Nathan Lambert
The canonical free textbook on RLHF and modern LLM post-training, from reward models to direct alignment.
Overview
The RLHF Book is a free, continuously updated web textbook by Nathan Lambert that organizes modern post-training around the canonical RLHF recipe, across 17 chapters and two appendices grouped into four parts. The introductory part covers a history of RLHF and how the training objective is actually constructed. The core training pipeline part walks through instruction fine-tuning, reward modeling from preference data, the reinforcement learning techniques used to optimize against those reward models, reasoning and inference-time scaling, direct-alignment algorithms such as DPO that skip the reward model entirely, and rejection sampling. A data-focused part covers the nature of human preferences, how preference data is collected in practice, the shift from human to synthetic data and model distillation, and training models for tool use and function calling. The final part is the practitioner's half that most RLHF material omits: over-optimization and why RLHF fails, regularization to constrain optimization to effective parameter regions, the evolving role of evaluation, and how labs craft model character to fit products. Appendices supply mathematical definitions for RL and language modeling plus an argument that RLHF does more than adjust style. The companion repository carries reference implementations for PPO, DPO and rejection sampling, dual-licensed CC-BY-NC-SA for prose and MIT for code. It exists as an arXiv preprint (2504.12501), a Manning print edition published July 2026, and a web version last updated August 2026, with a community Simplified Chinese translation.
At a Glance
- Topic
- Fine-Tuning
- Level
- Intermediate
- Format
- Book
- Cost
- Free
- Duration
- ~12-15 hours to read; 17 chapters plus 2 appendices, self-paced
- Provider
- Nathan Lambert
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Construct the RLHF training objective and reason about what it actually optimizes
- ✓Train a reward model from pairwise preference data as an RL optimization target
- ✓Implement and debug policy-gradient methods such as PPO for language models
- ✓Apply direct-alignment algorithms like DPO without training a separate reward model
- ✓Use rejection sampling and best-of-n as a cheaper alternative to full online RL
- ✓Design preference-data collection pipelines and judge when synthetic data suffices
- ✓Diagnose reward over-optimization and apply regularization to constrain policy drift
- ✓Post-train a model for tool use and function calling in its outputs
Highlights
- •The free web version stays current — last updated August 2026, a month after the July 2026 Manning print edition
- •Covers the recent shift to reasoning and inference-time-scaling RL, not just classic PPO-on-preferences
- •Companion GitHub repo ships reference implementations of PPO, DPO and rejection sampling under an MIT license
- •Chapters 14-17 are the practitioner's half most RLHF writing skips: over-optimization, regularization, evaluation, model character
- •Also published as an arXiv preprint (2504.12501), with a community Simplified Chinese translation
Who It’s For
Best For
- ✓Engineers building a post-training pipeline for an open-weight model
- ✓ML practitioners moving from supervised fine-tuning into RL-based alignment
- ✓Applied scientists who need to read RLHF papers critically rather than reimplement them
- ✓Teams deciding between DPO, rejection sampling and full online RL
Prerequisites
- •Entry-level familiarity with language modeling and transformer training
- •Basic reinforcement learning: policies, rewards, value functions, policy gradients
- •Comfort with mathematical notation — the book derives objectives rather than hand-waving them
FAQ
What is RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training?
A complete, free online textbook covering how modern language models are post-trained: instruction tuning, reward modeling, policy-gradient RL, direct alignment algorithms like DPO, rejection sampling, synthetic data, and evaluation. Written for engineers with entry-level language-modeling and reinforcement-learning experience, it gives you enough theory and reference code to implement a post-training pipeline or read the literature without getting lost.
Is RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training free?
RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training is free to access.
What level is RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training for?
RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training is aimed at a intermediate audience. Recommended background: Entry-level familiarity with language modeling and transformer training, Basic reinforcement learning: policies, rewards, value functions, policy gradients, Comfort with mathematical notation — the book derives objectives rather than hand-waving them.
How long does RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training take?
Expect roughly ~12-15 hours to read; 17 chapters plus 2 appendices, self-paced. Most learners work through it at their own pace.
What will I learn from RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training?
You'll learn: Construct the RLHF training objective and reason about what it actually optimizes; Train a reward model from pairwise preference data as an RL optimization target; Implement and debug policy-gradient methods such as PPO for language models; Apply direct-alignment algorithms like DPO without training a separate reward model; Use rejection sampling and best-of-n as a cheaper alternative to full online RL; Design preference-data collection pipelines and judge when synthetic data suffices; Diagnose reward over-optimization and apply regularization to constrain policy drift; Post-train a model for tool use and function calling in its outputs.
Topics
Sources
This page was written from 3 sources, 1 on domains other than rlhfbook.com.