The Smol Training Playbook: The Secrets to Building World-Class LLMs
by Hugging Face
Hugging Face's full, unvarnished account of training an LLM end to end — ablations, data curation, the training marathon, post-training and cluster debugging.
Overview
Published on October 30, 2025 by Loubna Ben Allal, Lewis Tunstall, Nouamane Tazi, Elie Bakouch, Ed Beeching, Carlos Miguel Patiño, Clémentine Fourrier, Thibaud Frere, Anton Lozhkov, Colin Raffel, Leandro von Werra and Thomas Wolf, The Smol Training Playbook is Hugging Face's follow-up to the Ultra-Scale Playbook and covers the full pipeline behind SmolLM3, a 3B-parameter multilingual reasoning model trained on 11 trillion tokens. It is organized as eight chapters: a 'Training Compass' framework for deciding why/what/how (and whether) to train at all; running small ablations before committing compute; model architecture decisions across attention mechanisms, embeddings, positional encoding and long-context approaches; the art of data curation and dataset mixing; optimizer and hyperparameter selection; 'The Training Marathon' on executing and scaling a long run; post-training via fine-tuning, alignment and model merging; and infrastructure, GPU cluster management and debugging. Its distinguishing value is candor about cost and failure — the authors note that ablations and debugging consumed more than half the cost of their main training run, making explicit the hidden research expense that published model reports usually omit.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- 8 chapters, self-paced (several hours)
- Provider
- Hugging Face
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Decide whether training (or post-training) your own model is justified at all, using the why/what/how Training Compass
- ✓Design and read small ablations so architecture and data choices are settled before you burn cluster time
- ✓Make concrete architecture decisions — attention variants, embeddings, positional encoding, long-context strategy
- ✓Curate and mix pre-training datasets, and tune optimizer and hyperparameter recipes at scale
- ✓Run a long multi-trillion-token training job: monitoring, instability, restarts and scaling
- ✓Post-train a base model with fine-tuning, alignment and model merging
- ✓Operate and debug GPU-cluster infrastructure for large training runs
Highlights
- •Written by the team that actually shipped SmolLM3 (3B params, 11T tokens) — not a survey
- •Documents what went wrong, including how much of the budget went to ablations and debugging
- •Free and readable in the browser, no signup or download
- •Companion to Hugging Face's Ultra-Scale Playbook, focused on the model-building decisions rather than only distributed-training mechanics
Who It’s For
Best For
- ✓ML engineers planning to pre-train or post-train an LLM
- ✓Research engineers running ablations and training infrastructure
- ✓Technical leads sizing the real cost and risk of an in-house model
- ✓AI engineers who want to understand how modern open models are actually built
Prerequisites
- •Solid deep-learning fundamentals (transformers, attention, optimizers)
- •PyTorch and distributed-training familiarity
- •Exposure to GPU cluster workflows helps
FAQ
What is The Smol Training Playbook: The Secrets to Building World-Class LLMs?
A free, book-length playbook from Hugging Face's pre-training team that walks through the entire lifecycle of building a language model, using their 3B-parameter SmolLM3 as the case study. Written for ML engineers and researchers who may actually train or post-train a model and need the decisions and failure modes, not just the final recipe.
Is The Smol Training Playbook: The Secrets to Building World-Class LLMs free?
The Smol Training Playbook: The Secrets to Building World-Class LLMs is free to access.
What level is The Smol Training Playbook: The Secrets to Building World-Class LLMs for?
The Smol Training Playbook: The Secrets to Building World-Class LLMs is aimed at a advanced audience. Recommended background: Solid deep-learning fundamentals (transformers, attention, optimizers), PyTorch and distributed-training familiarity, Exposure to GPU cluster workflows helps.
How long does The Smol Training Playbook: The Secrets to Building World-Class LLMs take?
Expect roughly 8 chapters, self-paced (several hours). Most learners work through it at their own pace.
What will I learn from The Smol Training Playbook: The Secrets to Building World-Class LLMs?
You'll learn: Decide whether training (or post-training) your own model is justified at all, using the why/what/how Training Compass; Design and read small ablations so architecture and data choices are settled before you burn cluster time; Make concrete architecture decisions — attention variants, embeddings, positional encoding, long-context strategy; Curate and mix pre-training datasets, and tune optimizer and hyperparameter recipes at scale; Run a long multi-trillion-token training job: monitoring, instability, restarts and scaling; Post-train a base model with fine-tuning, alignment and model merging; Operate and debug GPU-cluster infrastructure for large training runs.