Olmo 3
by Allen Institute for AI (Ai2)
The full recipe for a frontier-class open model — data, checkpoints, evals and RL, all released.
Overview
Olmo 3 is the technical report for the Allen Institute for AI's fully open language model family, posted to arXiv on 15 December 2025 as arXiv:2512.13961 by Team Olmo — 66 authors — revised in April 2026 and filed under cs.CL and cs.LG. Its claim is not merely open weights but the entire model flow: every stage, intermediate checkpoint, data point and dependency used to build the models is released, which is what lets a reader reproduce or audit a decision rather than take it on faith. The family spans 7B and 32B parameters across four variants: Olmo 3-Base, Olmo 3-Instruct, Olmo 3-Think (a reasoning model that emits intermediate thinking traces), and Olmo 3-RL Zero, a deliberately bootstrapped reinforcement-learning pathway intended to make RL algorithms benchmarkable against a clean, fully documented baseline. Pretraining uses Dolma 3, a roughly 9.3-trillion-token corpus of web pages, science PDFs processed with olmOCR, code, math and encyclopedic text, with a roughly 5.9T-token pretraining mix, a ~100B-token Dolmino mid-training mix drawn from a 2.2T pool, and a ~50B-token Longmino mix for long-context extension to around 65K tokens. Post-training runs supervised fine-tuning, DPO and reinforcement learning with verifiable rewards. The report also documents the surrounding toolchain released alongside it: Olmo-core for distributed training, open-instruct for post-training, OLMES for evaluation, decon for decontamination, datamap-rs and duplodocus for data cleaning, and OlmoTrace for tracing outputs back to training documents.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~2-4 hours to read the report; considerably longer if you follow the released checkpoints and code
- Provider
- Allen Institute for AI (Ai2)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓See a complete, publicly disclosed LLM training pipeline from raw corpus to RL post-training
- ✓Understand how Dolma 3's ~9.3T-token corpus was assembled, filtered and decontaminated
- ✓Follow the staged recipe: pretraining, Dolmino mid-training, then Longmino long-context extension
- ✓Compare SFT, DPO and RLVR post-training and what each stage actually contributed
- ✓Learn how a reasoning model is trained to produce and use intermediate thinking traces
- ✓Use the RL Zero variant as a clean baseline for benchmarking reinforcement-learning algorithms
- ✓Locate the released tooling — Olmo-core, open-instruct, OLMES, OlmoTrace, olmOCR — and what each does
Highlights
- •Genuinely reproducible: data, intermediate checkpoints, training code and evaluations are all released
- •Rare public detail on mid-training and long-context extension, stages most labs never describe
- •OlmoTrace lets you trace a model output back to training documents, though independent testers found it phrase-matching rather than semantic
- •RL Zero exists specifically so reinforcement-learning research has a clean, fully documented starting point
- •Independent reviewers note the training data is still largely web crawl, so 'fully open' means auditable, not licence-clean
Who It’s For
Best For
- ✓Researchers who need a fully documented reference training pipeline
- ✓Engineers planning a domain-specific pretraining or continued-pretraining run
- ✓Policy and procurement teams that need to audit model provenance
Prerequisites
- •Solid understanding of transformer pretraining, tokenisation and evaluation benchmarks
- •Familiarity with post-training methods such as SFT, DPO and RL from verifiable rewards
FAQ
What is Olmo 3?
The technical report behind Ai2's Olmo 3, a fully open 7B and 32B model family where the training data, every intermediate checkpoint, the evaluation suite and the training code are all published, not just the weights. It documents the Dolma 3 corpus, the pretraining, mid-training and long-context stages, the SFT/DPO/RLVR post-training recipe, and the Think and RL Zero reasoning variants.
Is Olmo 3 free?
Olmo 3 is free to access.
What level is Olmo 3 for?
Olmo 3 is aimed at a advanced audience. Recommended background: Solid understanding of transformer pretraining, tokenisation and evaluation benchmarks, Familiarity with post-training methods such as SFT, DPO and RL from verifiable rewards.
How long does Olmo 3 take?
Expect roughly ~2-4 hours to read the report; considerably longer if you follow the released checkpoints and code. Most learners work through it at their own pace.
What will I learn from Olmo 3?
You'll learn: See a complete, publicly disclosed LLM training pipeline from raw corpus to RL post-training; Understand how Dolma 3's ~9.3T-token corpus was assembled, filtered and decontaminated; Follow the staged recipe: pretraining, Dolmino mid-training, then Longmino long-context extension; Compare SFT, DPO and RLVR post-training and what each stage actually contributed; Learn how a reasoning model is trained to produce and use intermediate thinking traces; Use the RL Zero variant as a clean baseline for benchmarking reinforcement-learning algorithms; Locate the released tooling — Olmo-core, open-instruct, OLMES, OlmoTrace, olmOCR — and what each does.
Topics
Sources
This page was written from 4 sources, 3 on domains other than arxiv.org.