On-Policy Distillation
by Thinking Machines Lab
Post-train small LLMs with dense per-token teacher feedback, at a fraction of the compute RL needs.
Overview
On-Policy Distillation is an October 27, 2025 post on Thinking Machines Lab's Connectionism blog, written by Kevin Lu with collaborators at the lab. It argues that the two standard post-training options fail in opposite ways: off-policy distillation (supervised fine-tuning on teacher outputs) gives a dense signal but trains the student in contexts the teacher visits rather than the ones the student will reach, while reinforcement learning is on-policy but returns only one sparse reward per episode. On-policy distillation combines the two: the student samples trajectories and a teacher model scores each sampled token with a per-token reverse KL, which the authors describe as a one-line change on top of an existing RL implementation. The post has an implementation section (the reverse-KL loss, an illustration and pseudocode), a distillation-for-reasoning experiment, a distillation-for-personalization experiment, and a discussion of compute efficiency, data reuse and continual learning. In the reasoning experiment, a Qwen3-8B-Base student already at 60% on AIME'24 after 400K SFT examples reaches 70% after about 150 steps of on-policy distillation; the authors put that at 9x to roughly 30x less compute than extrapolating SFT to the same score, depending on what is counted. In the personalization experiment, mid-training Qwen3-8B on internal documents lifts internal-QA accuracy but drops IF-eval from 85% to 79%, and on-policy distillation recovers it to 83% while QA rises to 41%. Runnable recipes are in the open-source tinker-cookbook repository and run on the lab's Tinker training API.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~30 min read; optional hands-on recipes in tinker-cookbook take hours of training time
- Provider
- Thinking Machines Lab
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Why off-policy SFT and sparse-reward RL each fail differently during LLM post-training
- ✓How to compute a per-token reverse KL between student samples and teacher log-probabilities
- ✓How to turn an existing RL training loop into on-policy distillation with minimal code changes
- ✓How to compare post-training methods by total compute cost, not just by final benchmark score
- ✓How to distil math reasoning from a larger teacher into an 8B student model
- ✓How to add new domain knowledge by mid-training without losing instruction-following behaviour
- ✓How on-policy distillation can serve as a practical tool for continual learning after deployment
Highlights
- •Real compute accounting: a Qwen3-8B-Base student goes from 60% to 70% on AIME'24 in about 150 distillation steps, which the authors estimate at 9x to about 30x cheaper than extrapolating SFT
- •Cites the Qwen3 team's result that on-policy distillation reached 74.4% on AIME'24 at roughly one-tenth the GPU hours of RL, which reached 67.6% after 17,920 GPU hours
- •Includes a continual-learning experiment with numbers: IF-eval falls from 85% to 79% after mid-training on internal documents and recovers to 83% after on-policy distillation
- •Backed by runnable open-source recipes in thinking-machines-lab/tinker-cookbook (~4.1k GitHub stars), whose distillation README now targets Qwen3.5-9B models and adds multi-teacher and multi-turn tool-use variants
- •A July 2026 Hugging Face survey of distillation calls it the clearest practitioner version of the argument and lists DeepSeek-V4, GLM-5, MiMo-V2-Flash and Nemotron 3 Ultra among 2026 models that use on-policy distillation
Who It’s For
Best For
- ✓ML engineers running SFT or RL post-training who need a cheaper path to reasoning gains
- ✓Teams distilling a large teacher model into a small, cheaper model they can deploy
- ✓Practitioners adding domain knowledge to a chat model without breaking its instruction following
- ✓Researchers comparing distillation, SFT and RL on a fixed compute budget
Prerequisites
- •Working knowledge of LLM supervised fine-tuning and RL post-training methods such as PPO or GRPO
- •Comfort with KL divergence and per-token log-probabilities
- •To reproduce the results: Tinker API access, or your own RL stack plus a teacher model you can query for log-probs
FAQ
What is On-Policy Distillation?
A long-form technical write-up from Thinking Machines Lab explaining on-policy distillation, a post-training method in which a student LLM samples its own outputs and a stronger teacher grades every token. It is for ML engineers doing fine-tuning or RL post-training who want reasoning or domain behaviour in a smaller model without paying the full cost of reinforcement learning.
Is On-Policy Distillation free?
On-Policy Distillation is free to access.
What level is On-Policy Distillation for?
On-Policy Distillation is aimed at a advanced audience. Recommended background: Working knowledge of LLM supervised fine-tuning and RL post-training methods such as PPO or GRPO, Comfort with KL divergence and per-token log-probabilities, To reproduce the results: Tinker API access, or your own RL stack plus a teacher model you can query for log-probs.
How long does On-Policy Distillation take?
Expect roughly ~30 min read; optional hands-on recipes in tinker-cookbook take hours of training time. Most learners work through it at their own pace.
What will I learn from On-Policy Distillation?
You'll learn: Why off-policy SFT and sparse-reward RL each fail differently during LLM post-training; How to compute a per-token reverse KL between student samples and teacher log-probabilities; How to turn an existing RL training loop into on-policy distillation with minimal code changes; How to compare post-training methods by total compute cost, not just by final benchmark score; How to distil math reasoning from a larger teacher into an 8B student model; How to add new domain knowledge by mid-training without losing instruction-following behaviour; How on-policy distillation can serve as a practical tool for continual learning after deployment.
Topics
Sources
This page was written from 4 sources, 2 on domains other than thinkingmachines.ai.