Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
by Sebastian Raschka (Ahead of AI)
How Gemma 4, Laguna XS.2, ZAYA1-8B and DeepSeek V4 cut KV-cache and attention cost for long-context inference, explained one block change at a time.
Overview
This Ahead of AI article by Sebastian Raschka, published May 16, 2026, follows his Big LLM Architecture Comparison and his March 2026 visual guide to attention variants. It covers four open-weight model families chosen for how they reduce KV-cache size, memory traffic and attention FLOPs, the costs that dominate when reasoning models and agents keep long contexts. Section 1 explains cross-layer KV sharing in Gemma 4 E2B and E4B: E2B computes KV in 15 of 35 layers and E4B in 24 of 42, saving roughly 2.7 GB and 6 GB of cache at 128K context in bfloat16. Section 2 explains per-layer embeddings and why E2B has 2.3B effective parameters but 5.1B with embeddings. Section 3 covers Poolside's Laguna XS.2, which sets a different query-head count per layer through num_attention_heads_per_layer while keeping eight KV heads. Section 4 covers Zyphra's ZAYA1-8B and compressed convolutional attention, which runs attention inside the compressed latent space, unlike MLA. Section 5 covers DeepSeek V4: manifold-constrained hyper-connections with four residual streams and a reported 6.7% training overhead, and CSA and HCA compression along the sequence dimension, which the V4 paper says uses 10% of V3.2's KV cache at 1M tokens. Raschka notes where claims lack ablations. He links a pure-PyTorch Gemma 4 implementation in his LLMs-from-scratch repository (ch05/17_gemma4). The article closes by recommending that readers learn these changes one component at a time, starting from a GPT-2 block, and by promoting his book Build a Reasoning Model (From Scratch).
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~35 min read; longer if you work through the Gemma 4 from-scratch notebooks
- Provider
- Sebastian Raschka (Ahead of AI)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How cross-layer KV sharing in Gemma 4 E2B and E4B roughly halves the KV cache
- ✓How per-layer embeddings work and why Gemma 4 reports effective versus total parameter counts
- ✓How Laguna XS.2 varies query heads per layer while keeping the KV cache shape fixed
- ✓How compressed convolutional attention in ZAYA1-8B differs from DeepSeek's multi-head latent attention
- ✓How manifold-constrained hyper-connections widen the residual stream and keep training stable
- ✓How DeepSeek V4's CSA and HCA compress the KV cache along the sequence dimension
- ✓How to read a model's config.json to spot sliding-window ratios, head counts and KV sharing
- ✓Which of these efficiency claims are backed by ablations and which rest on vendor reports
Highlights
- •Gives concrete layer counts and cache sizes, such as 15 of 35 layers computing KV in Gemma 4 E2B, instead of describing techniques abstractly
- •Backed by a runnable pure-PyTorch Gemma 4 notebook with a KV-cache variant in the LLMs-from-scratch GitHub repository
- •Points out where papers lack ablations, for example declining to call CSA/HCA better than MLA in general
- •Discussed on Hacker News (35 points) and part of a series with the Big LLM Architecture Comparison and the LLM Architecture Gallery
Who It’s For
Best For
- ✓Inference engineers sizing GPU memory for long-context serving of open-weight models
- ✓ML engineers who implement or port model architectures in PyTorch
- ✓Researchers tracking how 2026 open-weight models changed the transformer block
Prerequisites
- •Solid understanding of the transformer block, KV caching and grouped-query attention
- •Familiarity with mixture-of-experts models and sliding-window attention; Raschka's earlier attention-variants guide covers these
FAQ
What is Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention?
Sebastian Raschka's May 16, 2026 article is an illustrated walkthrough of the architecture changes in recent open-weight LLMs that lower long-context cost: cross-layer KV sharing, per-layer embeddings, per-layer attention budgets, compressed convolutional attention, mHC residual streams and DeepSeek V4's CSA and HCA. It is for engineers who serve or fine-tune these models and want to read a config.json and know what changed.
Is Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention free?
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention is free to access.
What level is Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention for?
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention is aimed at a advanced audience. Recommended background: Solid understanding of the transformer block, KV caching and grouped-query attention, Familiarity with mixture-of-experts models and sliding-window attention; Raschka's earlier attention-variants guide covers these.
How long does Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention take?
Expect roughly ~35 min read; longer if you work through the Gemma 4 from-scratch notebooks. Most learners work through it at their own pace.
What will I learn from Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention?
You'll learn: How cross-layer KV sharing in Gemma 4 E2B and E4B roughly halves the KV cache; How per-layer embeddings work and why Gemma 4 reports effective versus total parameter counts; How Laguna XS.2 varies query heads per layer while keeping the KV cache shape fixed; How compressed convolutional attention in ZAYA1-8B differs from DeepSeek's multi-head latent attention; How manifold-constrained hyper-connections widen the residual stream and keep training stable; How DeepSeek V4's CSA and HCA compress the KV cache along the sequence dimension; How to read a model's config.json to spot sliding-window ratios, head counts and KV sharing; Which of these efficiency claims are backed by ablations and which rest on vendor reports.
Topics
Sources
This page was written from 3 sources, 2 on domains other than magazine.sebastianraschka.com.