gpt-oss-120b & gpt-oss-20b Model Card
by OpenAI
Full architecture, quantization and adversarial-safety disclosure for OpenAI's open-weight reasoning models.
Overview
OpenAI's gpt-oss model card, posted to arXiv on 8 August 2025 with more than 120 listed authors, documents the company's first open-weight language models since GPT-2. Both are autoregressive mixture-of-experts transformers: gpt-oss-120b has 36 layers, 128 experts and 116.8B total parameters with 5.1B active per token, while gpt-oss-20b has 24 layers, 32 experts and 20.9B total with 3.6B active; both route top-4. Attention alternates banded (128-token bandwidth) and dense patterns with 64 query heads over 8 key-value heads via grouped-query attention, rotary embeddings extended by YaRN to a 131,072-token context, and a learned bias in the softmax denominator acting as an attention sink. MoE weights are stored in MXFP4 at 4.25 bits per parameter, cutting checkpoints to 60.8 GiB and 12.8 GiB so the large model fits a single 80GB GPU and the small one runs in as little as 16GB. Training consumed 2.1 million H100-hours for the 120b, using the o200k_harmony tokenizer and a harmony chat format with analysis, commentary and final channels plus a System-Developer-User-Assistant-Tool instruction hierarchy. Reported results at high reasoning include 96.6% on AIME 2024 with tools, 80.1% GPQA Diamond, 90.0% MMLU, 62.4% SWE-Bench Verified and 2622 Codeforces Elo, alongside candid hallucination rates of 0.782 on SimpleQA.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- Technical report; ~60-90 min read
- Provider
- OpenAI
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓The MoE layout: 36 layers and 128 experts versus 24 and 32
- ✓How MXFP4 at 4.25 bits shrinks checkpoints to 60.8 and 12.8 GiB
- ✓Why alternating banded and dense attention keeps long context affordable
- ✓How YaRN extends rotary embeddings to a 131,072-token context window
- ✓What the o200k_harmony tokenizer and harmony channel format actually encode
- ✓How low, medium and high reasoning levels trade accuracy against latency
- ✓What adversarial worst-case fine-tuning found on biological, chemical and cyber risk
- ✓Where these models hallucinate, measured on SimpleQA and PersonQA
Highlights
- •Publishes adversarial worst-case fine-tuning results, rare in any model card
- •Full benchmark table spanning math, code, health and agentic tool use
- •Documents the harmony response format required to run the weights correctly
- •Apache-2.0 weights plus reference PyTorch, Triton and Metal implementations
- •Reports honest weaknesses including a 0.782 SimpleQA hallucination rate
Who It’s For
Best For
- ✓Engineers deciding whether to self-host an open-weight reasoning model
- ✓Safety researchers studying open-weight release risk methodology
- ✓Teams sizing GPU capacity for a 120B mixture-of-experts deployment
Prerequisites
- •Familiarity with transformer architecture and mixture-of-experts routing
- •Basic understanding of quantization and inference memory budgets
FAQ
What is gpt-oss-120b & gpt-oss-20b Model Card?
OpenAI's model card for gpt-oss-120b and gpt-oss-20b, its Apache-2.0 open-weight reasoning models. It documents the mixture-of-experts architecture, MXFP4 quantization, the harmony chat format required to run the weights, benchmark results across math, code, health and agentic tool use, and an unusually candid adversarial fine-tuning safety study. Written for engineers deciding whether to self-host.
Is gpt-oss-120b & gpt-oss-20b Model Card free?
gpt-oss-120b & gpt-oss-20b Model Card is free to access.
What level is gpt-oss-120b & gpt-oss-20b Model Card for?
gpt-oss-120b & gpt-oss-20b Model Card is aimed at a advanced audience. Recommended background: Familiarity with transformer architecture and mixture-of-experts routing, Basic understanding of quantization and inference memory budgets.
How long does gpt-oss-120b & gpt-oss-20b Model Card take?
Expect roughly Technical report; ~60-90 min read. Most learners work through it at their own pace.
What will I learn from gpt-oss-120b & gpt-oss-20b Model Card?
You'll learn: The MoE layout: 36 layers and 128 experts versus 24 and 32; How MXFP4 at 4.25 bits shrinks checkpoints to 60.8 and 12.8 GiB; Why alternating banded and dense attention keeps long context affordable; How YaRN extends rotary embeddings to a 131,072-token context window; What the o200k_harmony tokenizer and harmony channel format actually encode; How low, medium and high reasoning levels trade accuracy against latency; What adversarial worst-case fine-tuning found on biological, chemical and cyber risk; Where these models hallucinate, measured on SimpleQA and PersonQA.
Topics
Sources
This page was written from 3 sources, 2 on domains other than arxiv.org.