CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026)
by Stanford University (Chelsea Finn)
Chelsea Finn's Stanford deep RL course, from policy gradients to RL for LLM preference optimization and reasoning.
Overview
CS 224R: Deep Reinforcement Learning is Stanford's graduate course on algorithms that learn behaviour from experience with deep neural networks, taught in Spring 2026 by Prof. Chelsea Finn, a co-author of the Direct Preference Optimization (DPO) paper. The published schedule runs 17 lectures: course intro and MDPs, imitation learning, policy gradients, actor-critic methods, off-policy actor-critic, Q-learning, offline RL, reward learning, two lectures on RL for LLMs (preference optimization, then reasoning), model-based RL, multi-task and goal-conditioned RL, meta-RL, hierarchical RL and imitation learning, two lectures on RL for robots (sim-to-real transfer and RL for vision-language-action models), and a frontiers session. Coursework is three homeworks mixing conceptual questions with PyTorch coding problems (40%), a final project (35%) and an exam (25%). The default final project, 'RL Fine-Tuning of Language Models', has students implement the RL stack for LLM post-training by hand on the Countdown arithmetic reasoning task with a Qwen model: supervised fine-tuning as a warm start, pairwise preference optimization with a DPO/IPO-style objective, and online policy-gradient optimization with RLOO against a rule-based verifier reward, then a research extension such as off-policy sampling, test-time inference, tool-integrated reasoning or curriculum learning. High-level trainers like SFTTrainer are explicitly disallowed, so you write the losses yourself. CS229 or equivalent is a stated prerequisite. Lecture recordings from the Spring 2025 offering are public on YouTube and as a Stanford Online podcast.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- 10-week quarter: 17 lectures (~90 min each), 3 PyTorch homeworks and a final project; self-paced if following the public materials
- Provider
- Stanford University (Chelsea Finn)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Formulate problems as MDPs and train policies with imitation learning from demonstrations
- ✓Derive and implement policy gradient, actor-critic and off-policy actor-critic methods in PyTorch
- ✓Apply Q-learning and offline RL to learn from fixed datasets without new interaction
- ✓Learn reward functions from data and use them to shape agent behaviour
- ✓Use RL for LLM preference optimization and for improving LLM reasoning ability
- ✓Implement SFT, DPO/IPO-style preference optimization and RLOO with a verifier reward on Qwen
- ✓Understand model-based, multi-task, goal-conditioned, meta and hierarchical reinforcement learning methods
- ✓See how RL is applied to robots, including sim-to-real transfer and VLA fine-tuning
Highlights
- •Taught by Chelsea Finn, a co-author of the DPO paper, so the RL-for-LLM lectures come from the source
- •Default project makes you hand-write SFT, preference optimization and RLOO for a Qwen model on Countdown — no high-level trainers allowed
- •Covers both LLM post-training and robotics (sim-to-real, RL for vision-language-action models) in one syllabus
- •Previous-year lectures are public on YouTube and as a Stanford Online podcast, so self-learners can follow along
Who It’s For
Best For
- ✓ML engineers moving into LLM post-training who want the RL theory behind DPO, GRPO-style and RLOO methods
- ✓Researchers and grad-level learners wanting a rigorous deep RL foundation
- ✓Robotics and embodied-AI engineers working with imitation learning and VLAs
Prerequisites
- •Machine learning at the level of Stanford CS229
- •Deep learning and PyTorch proficiency
- •Comfort with probability, calculus and linear algebra
FAQ
What is CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026)?
CS 224R is Stanford's graduate deep reinforcement learning course, taught by Chelsea Finn in Spring 2026. It is for ML engineers who want the algorithms behind modern post-training: imitation learning, policy gradients, actor-critic, offline RL and reward learning, plus dedicated lectures on RL for LLM preference optimization and reasoning. Afterwards you can implement SFT, DPO-style preference optimization and RLOO for a language model.
Is CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026) free?
CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026) is free to access.
What level is CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026) for?
CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026) is aimed at a advanced audience. Recommended background: Machine learning at the level of Stanford CS229, Deep learning and PyTorch proficiency, Comfort with probability, calculus and linear algebra.
How long does CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026) take?
Expect roughly 10-week quarter: 17 lectures (~90 min each), 3 PyTorch homeworks and a final project; self-paced if following the public materials. Most learners work through it at their own pace.
What will I learn from CS 224R: Deep Reinforcement Learning (Stanford, Spring 2026)?
You'll learn: Formulate problems as MDPs and train policies with imitation learning from demonstrations; Derive and implement policy gradient, actor-critic and off-policy actor-critic methods in PyTorch; Apply Q-learning and offline RL to learn from fixed datasets without new interaction; Learn reward functions from data and use them to shape agent behaviour; Use RL for LLM preference optimization and for improving LLM reasoning ability; Implement SFT, DPO/IPO-style preference optimization and RLOO with a verifier reward on Qwen; Understand model-based, multi-task, goal-conditioned, meta and hierarchical reinforcement learning methods; See how RL is applied to robots, including sim-to-real transfer and VLA fine-tuning.
Topics
Sources
This page was written from 3 sources, 1 on domains other than cs224r.stanford.edu.