CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026)
by NYU Tandon School of Engineering (Pavel Izmailov)
A paper-by-paper tour of how frontier LLMs are built and post-trained with RL, from PPO and scaling laws to GRPO, DeepSeek-R1 and world models.
Overview
CS-GY 9223 L is a graduate seminar at NYU Tandon's Computer Science and Engineering department, run on Thursdays from September 2026 to December 2026 by Assistant Professor Pavel Izmailov, whose NYU faculty page lists research on reasoning in AI, reinforcement learning, planning and search, interpretability and technical AI alignment. The public course page carries the full schedule. It opens with LLM basics (Attention Is All You Need, GPT-2, GPT-3) and RL foundations (MDPs, policy gradients, PPO, read against Sutton and Barto and OpenAI Spinning Up), then moves to scaling laws (Kaplan, Chinchilla, data-constrained scaling, scaling RL compute), AlphaGo, AlphaZero and MuZero, attention variants and mixture-of-experts using DeepSeek-V3 and GQA, positional encoding and long context (RoPE, YaRN, ALiBi, RULER), and optimizers including Muon and muP. The second half covers RLHF and preference optimization (InstructGPT, DPO, Constitutional AI, Tulu 3), DeepSeek-R1 with the GRPO derivation from DeepSeekMath, reward hacking and over-optimization, weak-to-strong generalization, the Kimi K3 technical report, and world models (V-JEPA 2, DreamerV3, GameNGen, Genie 3). Lectures through October 29 have Google Slides decks and most have a Colab demo. Grading in the syllabus PDF is 20% reading responses, 40% a paper presentation and 40% a research or implementation project, with no exams, so for a self-learner the reading list and demos are the course.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- 13 weekly 2.5-hour sessions, Sep 10 to Dec 10, 2026, plus several assigned papers per session
- Provider
- NYU Tandon School of Engineering (Pavel Izmailov)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Derive policy gradients and PPO from MDP foundations before applying them to language models
- ✓Compare pre-training scaling laws from Kaplan and Chinchilla with newer results on data-constrained training and scaling RL compute
- ✓Take apart a frontier architecture: grouped-query attention, DeepSeek mixture-of-experts layers and long-context positional schemes like RoPE and YaRN
- ✓Evaluate optimizer choices such as Muon and muP and how they change large-model training dynamics
- ✓Work through RLHF, DPO and Constitutional AI using the InstructGPT and Tulu 3 papers as reference implementations
- ✓Follow the GRPO derivation and reinforcement learning with verifiable rewards as used in DeepSeek-R1 and Kimi k1.5
- ✓Recognize reward hacking and reward-model over-optimization, and the scaling laws that describe when it appears
- ✓Read and critique frontier technical reports such as Kimi K3 and world-model papers like V-JEPA 2 and DreamerV3
Highlights
- •The reading list is current to late 2026, including Muon, the Kimi K3 report and Genie 3, which older LLM courses do not reach
- •Ties classic RL (AlphaZero, MuZero, Sutton and Barto) directly to LLM post-training methods like GRPO and RLVR in one sequence
- •Most sessions through October 29 ship a Colab demo next to the slides, including RoPE and positional-encoding comparisons
- •Taught by an NYU professor whose own research covers reasoning, reinforcement learning and weak-to-strong generalization
- •The syllabus PDF is public, with the full prerequisites list and the project-based grading breakdown
Who It’s For
Best For
- ✓ML engineers moving into LLM post-training, RLHF or reasoning-model work
- ✓Graduate students and researchers who want a curated, current reading list on RL for language models
- ✓Engineers who already know transformers and want the paper trail behind DeepSeek-R1-style training
Prerequisites
- •A graduate-level machine learning or deep learning course (the syllabus names CS-GY 6923 or equivalent), with transformers strongly recommended
- •Probability and statistics, linear algebra including SVD, and calculus for gradients and backpropagation
- •Fluent Python and hands-on experience with PyTorch or JAX, since projects involve non-trivial implementation
FAQ
What is CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026)?
CS-GY 9223 is NYU Tandon's Fall 2026 graduate seminar on large language models, reinforcement learning and reasoning, taught by Pavel Izmailov. It suits engineers and researchers who already know deep learning and want to read the papers behind modern reasoning models, then run experiments on architecture choices, RLHF, DPO and GRPO.
Is CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026) free?
CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026) is free to access.
What level is CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026) for?
CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026) is aimed at a advanced audience. Recommended background: A graduate-level machine learning or deep learning course (the syllabus names CS-GY 6923 or equivalent), with transformers strongly recommended, Probability and statistics, linear algebra including SVD, and calculus for gradients and backpropagation, Fluent Python and hands-on experience with PyTorch or JAX, since projects involve non-trivial implementation.
How long does CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026) take?
Expect roughly 13 weekly 2.5-hour sessions, Sep 10 to Dec 10, 2026, plus several assigned papers per session. Most learners work through it at their own pace.
What will I learn from CS-GY 9223: Language Models, Reinforcement Learning, Reasoning (NYU, Fall 2026)?
You'll learn: Derive policy gradients and PPO from MDP foundations before applying them to language models; Compare pre-training scaling laws from Kaplan and Chinchilla with newer results on data-constrained training and scaling RL compute; Take apart a frontier architecture: grouped-query attention, DeepSeek mixture-of-experts layers and long-context positional schemes like RoPE and YaRN; Evaluate optimizer choices such as Muon and muP and how they change large-model training dynamics; Work through RLHF, DPO and Constitutional AI using the InstructGPT and Tulu 3 papers as reference implementations; Follow the GRPO derivation and reinforcement learning with verifiable rewards as used in DeepSeek-R1 and Kimi k1.5; Recognize reward hacking and reward-model over-optimization, and the scaling laws that describe when it appears; Read and critique frontier technical reports such as Kimi K3 and world-model papers like V-JEPA 2 and DreamerV3.
Topics
Sources
This page was written from 3 sources, 1 on domains other than izmailovpavel.github.io.