DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
by DeepSeek-AI
The architecture and systems detail behind the largest open-weight model yet, at a one-million-token context.
Overview
DeepSeek-V4 is a 2026 frontier-scale open-weight release documented in a six-section report: Introduction, Architecture, General Infrastructures, Pre-Training, Post-Training, and Conclusion/Limitations. The architecture section is the core of the paper — it carries forward DeepSeek's Mixture-of-Experts and multi-token-prediction designs and adds three new pieces: a hybrid attention stack combining Compressed Sparse Attention (CSA) with Heavily Compressed Attention (HCA) to make million-token context affordable, Manifold-Constrained Hyper-Connections (mHC) that replace conventional residual connections, and the Muon optimizer for faster and more stable convergence. The infrastructure section covers fine-grained expert parallelism with communication-computation overlap, kernel development in the TileLang DSL, and batch-invariant deterministic kernels — the operational detail most model reports omit. Pre-training used more than 32T tokens; post-training is a two-stage pipeline that first trains specialist models with SFT and RL, then consolidates them into one model through on-policy distillation. The headline efficiency result is that at a one-million-token context V4-Pro needs only 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2. Weights for both variants are MIT-licensed and published on Hugging Face, so the paper doubles as the reference manual for anyone deploying them. Simon Willison, writing on release day, placed V4-Pro roughly three to six months behind closed frontier models and noted V4-Flash pricing at $0.14 per million input tokens.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~90 min read; dense six-section technical report
- Provider
- DeepSeek-AI
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How Compressed Sparse Attention and Heavily Compressed Attention combine to make million-token context affordable
- ✓Why Manifold-Constrained Hyper-Connections replace conventional residual connections in a 1.6T-parameter Mixture-of-Experts stack
- ✓How the Muon optimizer changes convergence speed and training stability at frontier scale
- ✓The two-stage post-training recipe: specialist SFT plus RL, then on-policy distillation into one model
- ✓Fine-grained expert parallelism and communication-computation overlap techniques used to train both variants
- ✓How TileLang kernels and batch-invariant deterministic kernels support reproducible large-scale training and inference
- ✓The measured inference economics: 27% of V3.2 single-token FLOPs and 10% of its KV cache
Highlights
- •At 1.6T total parameters V4-Pro is the largest open-weight model released to date, under an MIT licence
- •Publishes the kernel and parallelism detail that most frontier model reports leave out entirely
- •V4-Flash at 284B total / 13B active is small enough to quantize onto a single high-memory workstation
- •Independently assessed on release day as roughly three to six months behind closed frontier models
- •Both variants' weights are on Hugging Face, so every claim in the paper is independently checkable
Who It’s For
Best For
- ✓Inference engineers evaluating long-context serving costs for open-weight models
- ✓Pre-training researchers designing attention variants and Mixture-of-Experts architectures
- ✓Platform teams deciding whether to self-host an open-weight frontier model
- ✓Anyone tracking how far behind the closed frontier open weights actually are
Prerequisites
- •Solid understanding of transformer architecture, attention variants and KV caching
- •Familiarity with Mixture-of-Experts routing and distributed training terminology
- •Comfort reading a dense, benchmark-heavy machine learning technical report
FAQ
What is DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence?
The official DeepSeek-V4 technical report, submitted to arXiv on 26 April 2026, documents how DeepSeek built two open-weight Mixture-of-Experts models — V4-Pro at 1.6T parameters (49B activated) and V4-Flash at 284B (13B activated) — that both serve a one-million-token context window. It is written for engineers who train, serve, or evaluate frontier-scale models and want the architecture and infrastructure detail rather than a launch summary.
Is DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence free?
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence is free to access.
What level is DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence for?
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence is aimed at a advanced audience. Recommended background: Solid understanding of transformer architecture, attention variants and KV caching, Familiarity with Mixture-of-Experts routing and distributed training terminology, Comfort reading a dense, benchmark-heavy machine learning technical report.
How long does DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence take?
Expect roughly ~90 min read; dense six-section technical report. Most learners work through it at their own pace.
What will I learn from DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence?
You'll learn: How Compressed Sparse Attention and Heavily Compressed Attention combine to make million-token context affordable; Why Manifold-Constrained Hyper-Connections replace conventional residual connections in a 1.6T-parameter Mixture-of-Experts stack; How the Muon optimizer changes convergence speed and training stability at frontier scale; The two-stage post-training recipe: specialist SFT plus RL, then on-policy distillation into one model; Fine-grained expert parallelism and communication-computation overlap techniques used to train both variants; How TileLang kernels and batch-invariant deterministic kernels support reproducible large-scale training and inference; The measured inference economics: 27% of V3.2 single-token FLOPs and 10% of its KV cache.
Topics
Sources
This page was written from 3 sources, 2 on domains other than arxiv.org.