Hy4-preview Model Card — Tencent's 770B Open-Weight MoE
by Tencent Hunyuan
A frontier-scale MoE card that publishes its sparse-attention design, its serving flags and its own known failure modes.
Overview
Hy4-preview is Tencent's new-generation Mixture-of-Experts flagship, released open-weight under a genuine Apache 2.0 licence on 28 August 2026 across Hugging Face, ModelScope, GitCode and CNB. The card documents 770B total parameters with 49B activated per token — an activation ratio just above 6 percent — across 78 backbone layers, the first a standard dense FFN and the remaining 77 MoE layers each holding 256 routed experts plus 1 shared expert, with top-8 routed experts plus the shared expert firing per token. Hidden size is 6144 with 64 attention heads, query compression at 2048 dimensions and KV compression at 512. Two architectural choices are named and worth studying: Gated DeepSeek Sparse Attention with an IndexCache for cross-layer sparse index reuse, and iHC (identity Hyper-Connections) to widen inter-layer information flow. Context length is 1M tokens over a 120,832-token vocabulary, and multi-token prediction ships for speculative decoding. On training, Tencent describes scaling model size, context length and training data together with a substantially larger post-training run, with data built around work shipped by internal software engineers, game developers, finance analysts and security experts. Evaluation is reported as a blind study by 163 internal experts across 203 engineering tasks — 2.99 average versus 2.92 for GLM 5.3 (46.8 percent wins) and 2.94 for Kimi K3 (51.2 percent wins) — alongside GPQA Diamond 92.3, SWE-bench Pro 65.7 percent and Apex Agents 37.1. Deployment sections give concrete vLLM (vllm/vllm-openai:hy4-preview) and SGLang (lmsysorg/sglang:hy4-preview) images with FP8, tensor parallelism of 8 and MTP/NEXTN speculation, plus FP8 and llama.cpp/Ollama/LM Studio/Jan quantisations and finetuning guidance.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~30-45 minute read; self-hosting the weights is a multi-GPU project measured in days
- Provider
- Tencent Hunyuan
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How a 770B MoE routes tokens through 256 experts plus a shared expert at top-8 sparsity
- ✓How Gated DeepSeek Sparse Attention reuses a cross-layer IndexCache to make 1M context affordable
- ✓What identity Hyper-Connections (iHC) are for and where they sit in the backbone
- ✓How query and KV compression dimensions trade memory against quality in a frontier-scale model
- ✓How to serve the weights with vLLM or SGLang using FP8, tensor parallelism and speculative decoding
- ✓How multi-token prediction is wired for MTP and NEXTN speculation at inference time
- ✓How a lab structures a blind expert evaluation, and why 163 internal raters is not third-party validation
- ✓Which quantisations exist for local use via llama.cpp, Ollama, LM Studio and Jan
Highlights
- •A real Apache 2.0 release with no monthly-active-user thresholds or custom licence clauses, unusual at this parameter count
- •Publishes exact serving recipes — container images, tensor-parallel degree, FP8 and speculation flags — so the numbers are reproducible rather than asserted
- •Names its own failure modes: spending longer than necessary reasoning through complex tasks and a tendency to over-verify its own work
- •Architectural detail is specific enough to learn from — routing topology, sparse-attention index reuse, compression dimensions and vocabulary size are all given
- •Read the benchmarks sceptically: TechNode notes the results are not independently verified and cover only selected engineering tasks, and the evaluation panel was internal to Tencent
Who It’s For
Best For
- ✓Engineers evaluating open-weight frontier models for self-hosted coding or agentic workloads
- ✓Inference and serving engineers who need concrete FP8 / tensor-parallel / speculative-decoding configurations
- ✓ML researchers studying sparse attention, MoE routing and long-context architecture choices
Prerequisites
- •Familiarity with Mixture-of-Experts routing and modern sparse-attention variants
- •Experience serving large models with vLLM or SGLang, including quantisation and tensor parallelism
- •Substantial multi-GPU hardware if you intend to run the weights rather than just read the card
FAQ
What is Hy4-preview Model Card — Tencent's 770B Open-Weight MoE?
The official model card for Hy4-preview, the 770B-parameter Mixture-of-Experts flagship Tencent's Hunyuan team open-weighted under Apache 2.0 on 28 August 2026. It is worth reading as a design document, not just a download page: it specifies the routing topology, the sparse-attention scheme, the 1M-token context configuration, blind-evaluation results against GLM 5.3 and Kimi K3, and reproducible vLLM and SGLang serving commands.
Is Hy4-preview Model Card — Tencent's 770B Open-Weight MoE free?
Hy4-preview Model Card — Tencent's 770B Open-Weight MoE is free to access.
What level is Hy4-preview Model Card — Tencent's 770B Open-Weight MoE for?
Hy4-preview Model Card — Tencent's 770B Open-Weight MoE is aimed at a advanced audience. Recommended background: Familiarity with Mixture-of-Experts routing and modern sparse-attention variants, Experience serving large models with vLLM or SGLang, including quantisation and tensor parallelism, Substantial multi-GPU hardware if you intend to run the weights rather than just read the card.
How long does Hy4-preview Model Card — Tencent's 770B Open-Weight MoE take?
Expect roughly ~30-45 minute read; self-hosting the weights is a multi-GPU project measured in days. Most learners work through it at their own pace.
What will I learn from Hy4-preview Model Card — Tencent's 770B Open-Weight MoE?
You'll learn: How a 770B MoE routes tokens through 256 experts plus a shared expert at top-8 sparsity; How Gated DeepSeek Sparse Attention reuses a cross-layer IndexCache to make 1M context affordable; What identity Hyper-Connections (iHC) are for and where they sit in the backbone; How query and KV compression dimensions trade memory against quality in a frontier-scale model; How to serve the weights with vLLM or SGLang using FP8, tensor parallelism and speculative decoding; How multi-token prediction is wired for MTP and NEXTN speculation at inference time; How a lab structures a blind expert evaluation, and why 163 internal raters is not third-party validation; Which quantisations exist for local use via llama.cpp, Ollama, LM Studio and Jan.
Topics
Sources
This page was written from 3 sources, 2 on domains other than huggingface.co.