The Synthetic Data Playbook: Generating Trillions of the Finest Tokens
by Hugging Face
Hugging Face's March 2026 write-up of 90 rephrasing experiments, 1T+ generated tokens and 12.7 GPU-years: how to generate synthetic pretraining data that works.
Overview
The Synthetic Data Playbook was published by Hugging Face on March 8, 2026. Its authors are Joel Niklaus, Guilherme Penedo, Hynek Kydlicek, Elie Bakouch, Lewis Tunstall, Ed Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra and Thomas Wolf, the team behind FineWeb and SmolLM. It asks what makes synthetic pretraining data from rephrased web pages useful, and answers with 90 rephrasing configurations, 333 train-and-evaluate runs, over 1 trillion generated tokens and 12.7 GPU-years of compute. The first section, Rephrasing the Web, defines the three axes of synthetic data, the source datasets (DCLM, FineWeb-Edu, Ultra-FineWeb, Nemotron-CC, Cosmopedia, REWIRE) and how success is measured. The Experiments section compares existing datasets, tests whether new prompts beat DCLM, and isolates the effect of the rephrasing model and the source data. Analyses covers seed variance, proxy-model size and per-benchmark effects. Synthetic data helps reading comprehension and math but not commonsense reasoning. Infrastructure benchmarks vLLM throughput with tensor parallelism, chunked prefill and n-gram speculative decoding across 100 parallel SLURM tasks. The final recipe rephrases FineWeb-Edu-HQ with SmolLM2-1.7B using four prompt formats (FAQ, math, table, tutorial) and mixes the output 70/30 with DCLM. The authors found that larger rephrasing models did not help. The resulting FinePhrase dataset (486B tokens) and the datatrove/nanotron/lighteval pipeline are open on the Hub and on GitHub.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~2-3 hour read, longer if you run the finephrase pipeline
- Provider
- Hugging Face
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Design rephrasing prompts (FAQ, math, table, tutorial) that turn web text into better pretraining data
- ✓Choose a rephrasing model size, and why a 1B-scale SmolLM2 matched much larger models
- ✓Pick mixing ratios between synthetic and human data, with 60-80% synthetic performing best
- ✓Evaluate synthetic data with small proxy models while controlling for seed variance and benchmark noise
- ✓Configure vLLM with tensor parallelism, chunked prefill and n-gram speculative decoding for bulk generation
- ✓Know which capabilities synthetic data improves (reading, math) and where it hurts commonsense reasoning
- ✓Recognise why standard quality scorers like DCLM-score and Edu-score mis-predict synthetic data value
Highlights
- •Large-scale open ablation study: 90 configurations, 333 training runs, 1T+ tokens and 12.7 GPU-years of compute
- •Ends with a reproducible recipe and releases the 486B-token FinePhrase dataset and the huggingface/finephrase pipeline code
- •Interactive charts, including a GPU throughput calculator for Qwen3, Gemma-3, GPT-OSS-120B and SmolLM2
- •Written by the FineWeb/SmolLM team, and states its open questions (scale beyond the tested regime, optimal ratios) plainly
Who It’s For
Best For
- ✓Researchers and engineers building pretraining or mid-training datasets for small and mid-size LLMs
- ✓Teams generating synthetic data at scale who need practical vLLM throughput settings
- ✓Practitioners following the Smol Training Playbook who want the data-side companion
Prerequisites
- •Solid understanding of LLM pretraining, tokens, and benchmark-based evaluation
- •Familiarity with batch LLM inference (vLLM) and multi-GPU or SLURM cluster workflows
- •Comfort reading ablation-style research results and charts
FAQ
What is The Synthetic Data Playbook: Generating Trillions of the Finest Tokens?
The Synthetic Data Playbook is a free interactive research guide from Hugging Face's FineWeb team for engineers building LLM pretraining or mid-training datasets. It reports 333 train-and-evaluate experiments on rephrasing web text with small LLMs and ends with a concrete recipe: prompts, rephrasing model, mixing ratio and vLLM inference setup. It also released the 486B-token FinePhrase dataset.
Is The Synthetic Data Playbook: Generating Trillions of the Finest Tokens free?
The Synthetic Data Playbook: Generating Trillions of the Finest Tokens is free to access.
What level is The Synthetic Data Playbook: Generating Trillions of the Finest Tokens for?
The Synthetic Data Playbook: Generating Trillions of the Finest Tokens is aimed at a advanced audience. Recommended background: Solid understanding of LLM pretraining, tokens, and benchmark-based evaluation, Familiarity with batch LLM inference (vLLM) and multi-GPU or SLURM cluster workflows, Comfort reading ablation-style research results and charts.
How long does The Synthetic Data Playbook: Generating Trillions of the Finest Tokens take?
Expect roughly ~2-3 hour read, longer if you run the finephrase pipeline. Most learners work through it at their own pace.
What will I learn from The Synthetic Data Playbook: Generating Trillions of the Finest Tokens?
You'll learn: Design rephrasing prompts (FAQ, math, table, tutorial) that turn web text into better pretraining data; Choose a rephrasing model size, and why a 1B-scale SmolLM2 matched much larger models; Pick mixing ratios between synthetic and human data, with 60-80% synthetic performing best; Evaluate synthetic data with small proxy models while controlling for seed variance and benchmark noise; Configure vLLM with tensor parallelism, chunked prefill and n-gram speculative decoding for bulk generation; Know which capabilities synthetic data improves (reading, math) and where it hurts commonsense reasoning; Recognise why standard quality scorers like DCLM-score and Edu-score mis-predict synthetic data value.
Topics
Sources
This page was written from 4 sources, 3 on domains other than huggingface.co.