On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
by Qwen Team, Alibaba
The Qwen team's own account of why Qwen3.8-Next is shaped the way it is — hybrid attention, gated residuals, and off-accelerator n-gram tables.
Overview
Submitted to arXiv on 31 August 2026 by the Qwen team at Alibaba (Zihan Qiu, Dayiheng Liu, Bo Zheng and more than thirty co-authors), this report documents the design of Qwen3.8-Next: a sparse mixture-of-experts model with 125B parameters, 6B activated per token, plus an additional 51B parameters of n-gram embedding tables deliberately held off the accelerator in host memory and prefetched during inference. Token mixing uses a layer-wise hybrid of Gated DeltaNet and global attention with one full-attention layer in every four; at the continued-pretraining stage the full-attention layers are swapped for Qwen Sparse Attention. The residual stream is widened through a Gated Residual design with four branches and elementwise gating. Rather than reporting one score, the paper evaluates each architectural component along three dimensions — loss, computational efficiency, and training stability — and reports how the optimal hyperparameters shift as a result. On results it is unusually specific about both directions: the model leads the 397B-A17B predecessor on eight of fourteen benchmarks and trails it on the remainder by at most 2.6 points, while using roughly one third the activated parameters, one third the training tokens, and about one ninth the training FLOPs. It is the reference to read alongside the Qwen3 technical report when reasoning about serving cost versus total memory footprint.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~90 min read (arXiv preprint, submitted 31 August 2026)
- Provider
- Qwen Team, Alibaba
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How a hybrid Gated DeltaNet and global-attention stack is laid out layer by layer
- ✓Why one full-attention layer in every four preserves quality at reduced cost
- ✓How Qwen Sparse Attention replaces full attention during the continued-pretraining stage
- ✓What a Gated Residual widens in the residual stream and why it helps stability
- ✓How 51B parameters of n-gram embeddings live off-accelerator and are prefetched
- ✓How to read MoE scaling claims stated in activated parameters, tokens and FLOPs
- ✓Which ablations judge loss, efficiency and training stability as one combined trade-off
- ✓Why activated parameters, not total parameters, drive inference cost for sparse models
Highlights
- •First-party architecture report from the Qwen team rather than a third-party recap or benchmark roundup
- •States wins and losses explicitly: leads the 397B-A17B predecessor on eight of fourteen benchmarks, trails by at most 2.6 points on the rest
- •Quantifies the cost side — roughly one third the activated parameters, one third the training tokens, one ninth the training FLOPs
- •Treats training stability as a first-class design axis alongside loss and throughput, which most model reports omit
- •Documents an unusual memory-hierarchy choice: keeping a 51B-parameter n-gram embedding table off the accelerator entirely
Who It’s For
Best For
- ✓Engineers evaluating whether to serve a sparse mixture-of-experts model in production
- ✓Researchers designing hybrid linear-attention and full-attention architectures
- ✓Inference engineers reasoning about activated parameters versus total memory footprint
- ✓Anyone tracking how current open-weight frontier architectures are actually constructed
Prerequisites
- •Solid grasp of transformer attention and mixture-of-experts routing
- •Comfort reading architecture ablation tables and scaling-style efficiency claims
- •Helpful: prior reading of the Qwen3 technical report for the baseline it compares against
FAQ
What is On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability?
This is the first-party architecture report for Qwen3.8-Next, a sparse mixture-of-experts model with 125B total parameters and 6B activated per token. It is written for engineers and researchers who need to understand how a current open-weight frontier model is actually assembled, and it justifies each design choice against three axes at once — loss, efficiency and training stability — rather than presenting a single benchmark table.
Is On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability free?
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability is free to access.
What level is On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability for?
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability is aimed at a advanced audience. Recommended background: Solid grasp of transformer attention and mixture-of-experts routing, Comfort reading architecture ablation tables and scaling-style efficiency claims, Helpful: prior reading of the Qwen3 technical report for the baseline it compares against.
How long does On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability take?
Expect roughly ~90 min read (arXiv preprint, submitted 31 August 2026). Most learners work through it at their own pace.
What will I learn from On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability?
You'll learn: How a hybrid Gated DeltaNet and global-attention stack is laid out layer by layer; Why one full-attention layer in every four preserves quality at reduced cost; How Qwen Sparse Attention replaces full attention during the continued-pretraining stage; What a Gated Residual widens in the residual stream and why it helps stability; How 51B parameters of n-gram embeddings live off-accelerator and are prefetched; How to read MoE scaling claims stated in activated parameters, tokens and FLOPs; Which ablations judge loss, efficiency and training stability as one combined trade-off; Why activated parameters, not total parameters, drive inference cost for sparse models.
Topics
Sources
This page was written from 2 sources, 1 on domains other than arxiv.org.