Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox
by Microsoft and Hugging Face
How to turn a benchmark's executable database-state checks into GRPO rewards for multi-turn tool-using agents, with the full training stack released.
Overview
Published on the Hugging Face blog on October 22, 2026, this tutorial comes from the Microsoft team behind ThinkingBox, a sandbox and benchmark for agents in stateful business workflows (arXiv 2608.19741, revised October 1, 2026). ThinkingBox-Bench has 507 policy-conditioned tasks across retail, hospitality, auto insurance, neobank IT and consulting IT/HR, and it grades each attempt by comparing the backend's final database state with the expected outcome, rejecting missing, wrong or extra changes. The post shows how those same executable checks become a binary 1 or 0 reward for GRPO, with an LLM judge used only for a minority of narrow rubric requirements. It walks through offline policy-relative difficulty selection (probing the base policy repeatedly, bucketing tasks at success boundaries of 0.08, 0.40, 0.60 and 0.90, and dropping tasks that are almost never or almost always solved), full-episode rollouts against isolated backends, and shared parsing code between training and evaluation. The worked result: Qwen3.8-27B trained for 50 GRPO steps on 187 tasks moved pass@1 from 51.70% to 60.78% and tasks passed on all twenty attempts from 38 to 107, while pass@20 dipped from 89.35% to 87.97%. A closing section covers installation, dry-run configuration, a three-node Ray launch and evaluation through OpenEnv with vLLM.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Tutorial
- Cost
- Free
- Duration
- Long-form tutorial plus open-source repo; config checks run without GPUs, the documented full training run uses 3 nodes of 8 H100s
- Provider
- Microsoft and Hugging Face
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Convert deterministic terminal-state database checks into binary RL rewards for multi-turn agents
- ✓Select a training pool by probing the base policy and bucketing tasks by estimated success
- ✓Explain why mixed-outcome rollout groups give GRPO a useful learning signal on agent tasks
- ✓Report both coverage (pass@k) and repeatability (all-20 passes) instead of pass@1 alone
- ✓Estimate cost per dependable task and cost per successful attempt for an agent campaign
- ✓Keep parser parity between training and evaluation so rewards and benchmark scores agree
- ✓Install the stack, apply the pinned VeRL 0.9.0 patches and dry-run the trainer before allocating GPUs
- ✓Evaluate a trained policy in the ThinkingBox environment through OpenEnv served with vLLM
Highlights
- •Releases everything needed to reproduce the run: microsoft/thinkingbox-training (MIT), the ThinkingBox framework at a pinned commit, and the benchmark data under CDLA-Permissive-2.0
- •Reports the trade-off honestly: repeatability rose in all five domains while pass@20 fell slightly, with auto insurance showing the largest breadth loss
- •Gives exact hyperparameters: token-mean GRPO, 18 groups of 8 rollouts (144 episodes per update), AdamW at 1e-6, zero weight decay, 50 steps on 24 H100s
- •Prices the result: cost per dependable task fell from $24.36 to $8.00 and cost per successful attempt from $0.177 to $0.139
- •The training repo is new and small (5 stars, 17 commits when checked on October 31, 2026), so expect to read code rather than rely on community answers
Who It’s For
Best For
- ✓ML engineers running RL post-training (GRPO) on tool-calling or customer-workflow agents
- ✓Teams building agent benchmarks who want their grading code to double as a reward function
- ✓Researchers studying reliability metrics such as pass^k for agents beyond single-attempt accuracy
- ✓Practitioners evaluating whether VeRL and OpenEnv fit their agent training pipeline
Prerequisites
- •Working knowledge of policy-gradient RL for LLMs, especially GRPO and rollout groups
- •Python 3.12, uv, Ray and vLLM familiarity; Linux with NVIDIA GPUs and CUDA/NCCL for real training
- •Access to multi-node H100 hardware to reproduce the documented full-parameter Qwen3.8-27B run
- •Basic understanding of MCP-style tool calling and multi-turn agent episodes
FAQ
What is Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox?
A Microsoft and Hugging Face tutorial for ML engineers who post-train tool-using agents with reinforcement learning. It shows how ThinkingBox reuses deterministic checks on final database state as binary RL rewards, how to select tasks by difficulty before training, and how to run the released VeRL training stack yourself.
Is Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox free?
Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox is free to access.
What level is Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox for?
Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox is aimed at a advanced audience. Recommended background: Working knowledge of policy-gradient RL for LLMs, especially GRPO and rollout groups, Python 3.12, uv, Ray and vLLM familiarity; Linux with NVIDIA GPUs and CUDA/NCCL for real training, Access to multi-node H100 hardware to reproduce the documented full-parameter Qwen3.8-27B run, Basic understanding of MCP-style tool calling and multi-turn agent episodes.
How long does Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox take?
Expect roughly Long-form tutorial plus open-source repo; config checks run without GPUs, the documented full training run uses 3 nodes of 8 H100s. Most learners work through it at their own pace.
What will I learn from Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox?
You'll learn: Convert deterministic terminal-state database checks into binary RL rewards for multi-turn agents; Select a training pool by probing the base policy and bucketing tasks by estimated success; Explain why mixed-outcome rollout groups give GRPO a useful learning signal on agent tasks; Report both coverage (pass@k) and repeatability (all-20 passes) instead of pass@1 alone; Estimate cost per dependable task and cost per successful attempt for an agent campaign; Keep parser parity between training and evaluation so rewards and benchmark scores agree; Install the stack, apply the pinned VeRL 0.9.0 patches and dry-run the trainer before allocating GPUs; Evaluate a trained policy in the ThinkingBox environment through OpenEnv served with vLLM.
Topics
Sources
This page was written from 3 sources, 2 on domains other than huggingface.co.