LoRA Without Regret
by Thinking Machines Lab (John Schulman et al.)
Thinking Machines' controlled study of when LoRA matches full fine-tuning, and the rank, learning-rate and layer settings that make it happen.
Overview
LoRA Without Regret was published on September 29, 2025 on Thinking Machines Lab's Connectionism blog by John Schulman and colleagues, and it has a citable DOI (10.64434/tml.20250929). It asks a practical question: when does low-rank adaptation (LoRA) match full fine-tuning (FullFT)? It answers with sweeps over supervised fine-tuning on Tulu3 and OpenThoughts3 and reinforcement learning on MATH, GSM8K and DeepMath, using Llama 3 and Qwen3 models. The Methods and Results section covers four findings. On rank, high-rank LoRA tracks FullFT until the adapter runs out of capacity for the dataset. On batch size, LoRA pays a larger loss penalty at large batch sizes, and more rank does not fix it. On layers, attention-only LoRA underperforms, so the MLP and MoE layers must be included. On RL, even rank-1 LoRA matches FullFT, because policy-gradient training takes in only about one bit of information per episode. The Setting LoRA Hyperparameters section shows that the 1/r scaling makes the optimal learning rate roughly independent of rank, and that only two of LoRA's four hyperparameters are independent. It also finds the optimal LoRA learning rate is consistently about 10x the FullFT rate, closer to 15x for runs of about 100 steps. The Discussion explains the all-layers result through the empirical neural tangent kernel, estimates how much capacity SFT and RL need, and notes that LoRA uses about two-thirds of FullFT's FLOPs per training pass. It ends with open questions, including LoRA variants such as PiSSA and LoRA for mixture-of-experts models. Hugging Face's TRL documentation publishes an official reproduction guide with SFTTrainer and GRPOTrainer recipes.
At a Glance
- Topic
- Fine-Tuning
- Level
- Intermediate
- Format
- Paper
- Cost
- Free
- Duration
- ~1 hour read; the Hugging Face TRL reproduction scripts run as single-A100 jobs of up to 4-8 hours
- Provider
- Thinking Machines Lab (John Schulman et al.)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓When LoRA matches full fine-tuning, and how to tell when a dataset exceeds the adapter's capacity
- ✓Why applying LoRA to all layers, especially MLP and MoE layers, beats attention-only LoRA at matched parameter count
- ✓How to set the LoRA learning rate at roughly 10x the full fine-tuning rate, and about 15x for short runs
- ✓Why the 1/r scaling factor makes the optimal learning rate nearly independent of LoRA rank
- ✓Why LoRA is less tolerant of large batch sizes than full fine-tuning, and why raising rank does not fix it
- ✓Why rank-1 LoRA can match full fine-tuning in reinforcement learning, using a bits-per-episode capacity argument
- ✓How LoRA's roughly two-thirds FLOPs per training pass changes the compute budget versus full fine-tuning
- ✓How to turn the findings into TRL configs: rank 256 for post-training-scale SFT, rank 1-32 for RL, all-linear target modules
Highlights
- •Written by John Schulman and the Thinking Machines Lab team, published with a citable DOI so it can be referenced like a paper
- •Controlled experiments across SFT and RL on open datasets (Tulu3, OpenThoughts3, MATH, GSM8K, DeepMath) and open models (Llama 3, Qwen3)
- •Gives defaults you can apply today (target all layers, ~10x learning rate, small effective batch), and Hugging Face reproduced them with SmolLM3-3B in an official TRL guide
- •Widely discussed on Hacker News: 184 points and 58 comments on the Sep 29, 2025 thread (checked 2026-09-22)
- •Honest caveat: HN commenters faulted its thin engagement with prior LoRA-vs-full-fine-tuning research (e.g. 'intruder dimensions') and questioned the one-bit-per-episode RL argument. Read the theory as a hypothesis; the hyperparameter findings are the useful part
Who It’s For
Best For
- ✓ML engineers fine-tuning open-weight LLMs with LoRA on a limited GPU budget
- ✓Teams deciding between LoRA and full fine-tuning for SFT or RL post-training
- ✓Practitioners running GRPO or other RL fine-tuning who want to cut memory with low-rank adapters
- ✓Researchers studying the training dynamics of parameter-efficient fine-tuning
Prerequisites
- •Hands-on experience fine-tuning a model with LoRA, for example via PEFT or TRL
- •Comfort reading learning-rate sweeps, batch-size effects and loss curves
- •Basic familiarity with SFT versus RL post-training; the neural-tangent-kernel discussion is optional
FAQ
What is LoRA Without Regret?
LoRA Without Regret is a research write-up from Thinking Machines Lab, led by John Schulman, for engineers who fine-tune LLMs with LoRA. Through controlled supervised and reinforcement-learning experiments, it shows when LoRA matches full fine-tuning, so you can pick ranks, learning rates, target layers and batch sizes without guesswork.
Is LoRA Without Regret free?
LoRA Without Regret is free to access.
What level is LoRA Without Regret for?
LoRA Without Regret is aimed at a intermediate audience. Recommended background: Hands-on experience fine-tuning a model with LoRA, for example via PEFT or TRL, Comfort reading learning-rate sweeps, batch-size effects and loss curves, Basic familiarity with SFT versus RL post-training; the neural-tangent-kernel discussion is optional.
How long does LoRA Without Regret take?
Expect roughly ~1 hour read; the Hugging Face TRL reproduction scripts run as single-A100 jobs of up to 4-8 hours. Most learners work through it at their own pace.
What will I learn from LoRA Without Regret?
You'll learn: When LoRA matches full fine-tuning, and how to tell when a dataset exceeds the adapter's capacity; Why applying LoRA to all layers, especially MLP and MoE layers, beats attention-only LoRA at matched parameter count; How to set the LoRA learning rate at roughly 10x the full fine-tuning rate, and about 15x for short runs; Why the 1/r scaling factor makes the optimal learning rate nearly independent of LoRA rank; Why LoRA is less tolerant of large batch sizes than full fine-tuning, and why raising rank does not fix it; Why rank-1 LoRA can match full fine-tuning in reinforcement learning, using a bits-per-episode capacity argument; How LoRA's roughly two-thirds FLOPs per training pass changes the compute budget versus full fine-tuning; How to turn the findings into TRL configs: rank 256 for post-training-scale SFT, rank 1-32 for RL, all-linear target modules.
Topics
Sources
This page was written from 3 sources, 2 on domains other than thinkingmachines.ai.