Gemma 4 Technical Report
by Gemma Team, Google DeepMind
The primary source on Google's Apache-2.0 multimodal open-weight family, 2.3B to 31B.
Overview
Submitted to arXiv on 2 July 2026 (revised 24 July 2026) under cs.CL and cs.AI, this 17-page report from the Gemma Team at Google DeepMind — 321 listed authors — documents the fourth generation of Google's open-weight model family. The suite spans dense and Mixture-of-Experts architectures from 2.3B effective parameters (E2B) up to a 30.7B dense 31B model, including a 26B MoE variant with roughly 3.8B active parameters, and every checkpoint ships under Apache 2.0 rather than a restricted community licence. The report's substantive contributions are architectural: a hybrid attention scheme interleaving local sliding-window attention with full global attention to make long context affordable at inference time; a unified, encoder-free architecture for the 12B model that ingests raw audio and image patches directly instead of routing them through separate modality encoders; and an integrated thinking mode that lets the models emit reasoning traces before answering. Context windows are 128K tokens for the smallest E2B and E4B variants and 256K for the medium and larger ones, with multilingual coverage of more than 140 languages. All sizes handle text and variable-resolution images, while E2B, E4B and 12B add audio for speech recognition and translation. The paper reports gains across STEM, multimodal and long-context benchmarks — the 31B instruction-tuned model scores 85.2% on MMLU Pro, 89.2% on AIME 2026 and 84.3% on GPQA Diamond — and argues the family rivals larger frontier open models in human-rated tasks. It pairs directly with the five published Hugging Face model cards.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- 17 pages, ~45-60 minute read
- Provider
- Gemma Team, Google DeepMind
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How Gemma 4 mixes dense and Mixture-of-Experts architectures across a 2.3B to 31B range
- ✓Why hybrid local sliding-window plus global attention keeps long-context inference affordable
- ✓How the 12B model drops separate modality encoders and ingests raw audio and image patches
- ✓What a built-in thinking mode changes about training and inference-time reasoning traces
- ✓The design choices behind a 256K-token context window in a model you can self-host
- ✓How Gemma 4 scores on STEM, multimodal and long-context benchmarks against larger open models
- ✓Which checkpoint to pick for on-device versus server deployment under an Apache 2.0 licence
Highlights
- •Primary source for weights you can actually download and ship — Apache 2.0, not a restricted community licence
- •Authored by the Gemma Team at Google DeepMind with 321 listed contributors
- •17 pages, so it reads in a single sitting rather than as a 60-page frontier system card
- •Documents an encoder-free multimodal design that is still unusual among open-weight models
- •Maps one-to-one onto the five published Hugging Face checkpoints, so the numbers are reproducible
Who It’s For
Best For
- ✓Engineers choosing an open-weight model to self-host for multimodal workloads
- ✓ML practitioners tracking Mixture-of-Experts and long-context architecture decisions
- ✓Teams evaluating on-device or edge deployment of a small multimodal model
Prerequisites
- •Working knowledge of transformer architecture and attention variants
- •Familiarity with LLM benchmark names such as MMLU Pro, GPQA Diamond and AIME
FAQ
What is Gemma 4 Technical Report?
The primary source on Google DeepMind's Gemma 4 family of open-weight, natively multimodal language models — checkpoints from 2.3B to 31B parameters under Apache 2.0, spanning dense and Mixture-of-Experts variants, with a built-in thinking mode and context windows up to 256K tokens. Read this before picking an open model to self-host: it explains the architecture trade-offs a benchmark table alone will not tell you.
Is Gemma 4 Technical Report free?
Gemma 4 Technical Report is free to access.
What level is Gemma 4 Technical Report for?
Gemma 4 Technical Report is aimed at a advanced audience. Recommended background: Working knowledge of transformer architecture and attention variants, Familiarity with LLM benchmark names such as MMLU Pro, GPQA Diamond and AIME.
How long does Gemma 4 Technical Report take?
Expect roughly 17 pages, ~45-60 minute read. Most learners work through it at their own pace.
What will I learn from Gemma 4 Technical Report?
You'll learn: How Gemma 4 mixes dense and Mixture-of-Experts architectures across a 2.3B to 31B range; Why hybrid local sliding-window plus global attention keeps long-context inference affordable; How the 12B model drops separate modality encoders and ingests raw audio and image patches; What a built-in thinking mode changes about training and inference-time reasoning traces; The design choices behind a 256K-token context window in a model you can self-host; How Gemma 4 scores on STEM, multimodal and long-context benchmarks against larger open models; Which checkpoint to pick for on-device versus server deployment under an Apache 2.0 licence.
Topics
Sources
This page was written from 2 sources, 1 on domains other than arxiv.org.