Kimi K3: Open Frontier Intelligence
by Moonshot AI (Kimi Team)
How the first open 3T-class model was built — hybrid attention, 896-expert MoE, and million-token agentic RL.
Overview
Kimi K3 is Moonshot AI's technical report for the first open 3T-class model: a 2.8-trillion-parameter Mixture-of-Experts with 104 billion activated parameters, native vision, and a one-million-token context window. It was submitted to arXiv on 27 July 2026 (revised 7 August) under cs.CL and cs.LG, with over 400 listed authors. Architecturally the model is built on Kimi Delta Attention and Attention Residuals — mechanisms for improving information flow across sequence length and model depth — interleaved as 69 KDA and 24 Gated MLA layers across 93 total layers, alongside a 401-million-parameter vision encoder. Stable LatentMoE effectively activates 16 of 896 routed experts per token, and together with refined training and data recipes the authors report roughly a 2.5x improvement in overall scaling efficiency over Kimi K2. The post-training chapters are the ones agent builders should read closely: reinforcement learning across general, agentic and coding domains at multiple reasoning-effort levels, aimed at compositional generalisation and long-horizon execution, supported by million-token agentic RL infrastructure with persistent rollout and sandbox states. Infrastructure sections cover algorithm-system co-design for KDA, perfectly balanced expert-parallel training with explicit memory management, and deployment. Reported results include 88.3 on Terminal-Bench 2.1 and 81.2 on FrontierSWE. The authors state plainly that K3 still trails the strongest proprietary models, Claude Fable 5 and GPT-5.6 Sol, while beating every other model in their suite. Weights are released and run under vLLM, SGLang and TokenSpeed.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~3 hours for a careful read of the full technical report
- Provider
- Moonshot AI (Kimi Team)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How Kimi Delta Attention and Attention Residuals move information across depth and sequence
- ✓Why Stable LatentMoE activates only 16 of 896 experts and what that buys
- ✓How agentic reinforcement learning is staged across general, coding and tool-use domains
- ✓What million-token agentic RL requires: persistent rollout and sandbox state management
- ✓How expert-parallel training stays balanced and memory-managed at 2.8 trillion parameters
- ✓Where an open 3T-class model still trails leading proprietary models, and by how much
- ✓How multiple reasoning-effort levels are trained into one deployed model
- ✓What serving K3 in practice takes across vLLM, SGLang and TokenSpeed
Highlights
- •The first open 3T-class model shipped with a full technical report rather than a model card
- •Quantifies its own architecture win: roughly 2.5x scaling efficiency over Kimi K2
- •Unusually detailed on RL infrastructure — million-token rollouts with persistent sandbox state
- •States openly where it loses to Claude Fable 5 and GPT-5.6 Sol instead of cherry-picking
- •Weights are released under the Kimi K3 License, so the paper's claims are independently checkable
Who It’s For
Best For
- ✓Engineers evaluating open-weight frontier models for self-hosted deployment
- ✓Researchers working on MoE sparsity, hybrid attention or long-context architectures
- ✓Teams designing agentic RL training environments and execution sandboxes
- ✓Anyone tracking the open-versus-proprietary capability gap with concrete numbers
Prerequisites
- •Solid grasp of transformer architecture, attention variants and Mixture-of-Experts routing
- •Familiarity with LLM post-training: SFT, reward design and RL fine-tuning
- •Helpful but optional: prior reading of the Kimi K2 report as the comparison baseline
FAQ
What is Kimi K3: Open Frontier Intelligence?
The Kimi K3 technical report documents how Moonshot AI trained the first openly released 3-trillion-parameter-class model, covering its hybrid attention design, sparse Mixture-of-Experts routing, agentic reinforcement-learning post-training and the serving infrastructure underneath. It is written for engineers and researchers who want the mechanism rather than the launch post. Read it to understand where frontier open-weight capability now comes from — sparsity and RL environment design more than brute-force parameter scaling — and to judge whether K3 is worth self-hosting.
Is Kimi K3: Open Frontier Intelligence free?
Kimi K3: Open Frontier Intelligence is free to access.
What level is Kimi K3: Open Frontier Intelligence for?
Kimi K3: Open Frontier Intelligence is aimed at a advanced audience. Recommended background: Solid grasp of transformer architecture, attention variants and Mixture-of-Experts routing, Familiarity with LLM post-training: SFT, reward design and RL fine-tuning, Helpful but optional: prior reading of the Kimi K2 report as the comparison baseline.
How long does Kimi K3: Open Frontier Intelligence take?
Expect roughly ~3 hours for a careful read of the full technical report. Most learners work through it at their own pace.
What will I learn from Kimi K3: Open Frontier Intelligence?
You'll learn: How Kimi Delta Attention and Attention Residuals move information across depth and sequence; Why Stable LatentMoE activates only 16 of 896 experts and what that buys; How agentic reinforcement learning is staged across general, coding and tool-use domains; What million-token agentic RL requires: persistent rollout and sandbox state management; How expert-parallel training stays balanced and memory-managed at 2.8 trillion parameters; Where an open 3T-class model still trails leading proprietary models, and by how much; How multiple reasoning-effort levels are trained into one deployed model; What serving K3 in practice takes across vLLM, SGLang and TokenSpeed.
Topics
Sources
This page was written from 3 sources, 2 on domains other than arxiv.org.