The Smol Training Playbook: The Secrets to Building World-Class LLMs
by Hugging Face
The full, unglamorous record of how SmolLM3 was trained — including the parts that went wrong.
Overview
Published 30 October 2025 by twelve Hugging Face authors including Loubna Ben Allal, Lewis Tunstall, Nouamane Tazi, Elie Bakouch, Clémentine Fourrier, Colin Raffel, Leandro von Werra and Thomas Wolf, the playbook documents how SmolLM3 — a 3B-parameter multilingual reasoning model trained on 11T tokens — was actually built. The structure runs: a training compass that forces the why → what → how question and argues most teams should not pretrain at all; a chapter on ablations, covering baseline choice, derisking modifications, framework selection, evaluation and how to cost an ablation programme; architecture, covering attention variants, embedding sharing, positional encodings and long context, stability, MoE versus dense, hybrid models and the tokenizer, followed by optimisers beyond AdamW, learning rate, batch size and scaling laws; data curation, on why mixtures are unintuitive, how to test recipes systematically, and the specific English web, multilingual, code and math sources used; the training marathon, with a pre-flight checklist and three diagnosed production mysteries — vanishing throughput, persisting throughput drops and noisy loss — plus mid-training stage mixtures and long-context extension from 4k to 128k tokens; post-training, covering SFT, chat templates, preference optimisation and applying RLVR to hybrid reasoning models; and an infrastructure chapter on GPU internals, the roofline model, NVLink, EFA and libfabric interconnects, node health monitoring, checkpoint management and a three-step method for choosing a parallelism configuration. The authors state a reading time of two to four days. It closes a series that also produced FineWeb, the Ultra-Scale Playbook and the Evaluation Guidebook.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- Reading time stated by the authors: 2-4 days
- Provider
- Hugging Face
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Decide honestly whether training from scratch is justified rather than fine-tuning
- ✓Stand up an ablation pipeline and estimate what the ablation programme will cost
- ✓Choose between dense, MoE and hybrid architectures with the tradeoffs made explicit
- ✓Design a multi-stage data mixture across web, multilingual, code and math sources
- ✓Diagnose throughput collapse, loss spikes and noisy curves during a live run
- ✓Extend context length from 4k to 128k tokens as a mid-training stage
- ✓Sequence a post-training pipeline through SFT, preference optimisation and RLVR
- ✓Pick a parallelism configuration by fitting memory, batch size, then throughput
Highlights
- •Documents a real model end to end rather than describing the technique in the abstract
- •Unusually candid about failure: restarts, debugging and compute spent on dead ends
- •The infrastructure chapter covers interconnects and node health that most training write-ups skip
- •Each major chapter ends with explicit 'rules of engagement' you can apply directly
- •Written by the team behind FineWeb, the Ultra-Scale Playbook and the Evaluation Guidebook
Who It’s For
Best For
- ✓Teams seriously considering pretraining a model from scratch
- ✓ML engineers building domain-specific or deployment-constrained models
- ✓Infrastructure engineers running multi-node GPU training clusters
Prerequisites
- •Solid transformer fundamentals — attention, tokenization, optimisation
- •Experience training or fine-tuning models on multi-GPU hardware
- •Comfort reading loss curves, evaluation benchmarks and throughput metrics
FAQ
What is The Smol Training Playbook: The Secrets to Building World-Class LLMs?
A book-length engineering account of pretraining and post-training a modern LLM, written by the Hugging Face team that built SmolLM3. It walks the whole arc — deciding whether to train at all, running ablations, choosing architecture, mixing data, surviving the training run, post-training, and the GPU infrastructure underneath — and is unusually candid about failures, restarts and what the compute actually got spent on.
Is The Smol Training Playbook: The Secrets to Building World-Class LLMs free?
The Smol Training Playbook: The Secrets to Building World-Class LLMs is free to access.
What level is The Smol Training Playbook: The Secrets to Building World-Class LLMs for?
The Smol Training Playbook: The Secrets to Building World-Class LLMs is aimed at a advanced audience. Recommended background: Solid transformer fundamentals — attention, tokenization, optimisation, Experience training or fine-tuning models on multi-GPU hardware, Comfort reading loss curves, evaluation benchmarks and throughput metrics.
How long does The Smol Training Playbook: The Secrets to Building World-Class LLMs take?
Expect roughly Reading time stated by the authors: 2-4 days. Most learners work through it at their own pace.
What will I learn from The Smol Training Playbook: The Secrets to Building World-Class LLMs?
You'll learn: Decide honestly whether training from scratch is justified rather than fine-tuning; Stand up an ablation pipeline and estimate what the ablation programme will cost; Choose between dense, MoE and hybrid architectures with the tradeoffs made explicit; Design a multi-stage data mixture across web, multilingual, code and math sources; Diagnose throughput collapse, loss spikes and noisy curves during a live run; Extend context length from 4k to 128k tokens as a mid-training stage; Sequence a post-training pipeline through SFT, preference optimisation and RLVR; Pick a parallelism configuration by fitting memory, batch size, then throughput.
Topics
Sources
This page was written from 3 sources, 2 on domains other than huggingface.co.