LLM Architecture Gallery
by Sebastian Raschka
98 LLM architectures normalised onto the same card, with a side-by-side diff tool.
Overview
Sebastian Raschka — author of 'Build a Large Language Model (From Scratch)' and 'Build a Reasoning Model (From Scratch)' — maintains this side-by-side reference covering 98 text-focused LLM architectures and language-model backbones, with entries running through mid-2026. It exists to answer a question that papers make expensive: what actually changed between these models. Each entry is a card carrying an architecture diagram, release date, parameter count, context window and license, the decoder type (dense, sparse mixture-of-experts, hybrid or recurrent), the attention variant (MHA, GQA, MLA and so on), a layer-composition breakdown, the KV-cache memory required per token, a short note on the model's distinguishing feature, and where available an AA Intelligence Index score. Entries link out to the model's config.json on Hugging Face, its technical report, and for many models a from-scratch PyTorch implementation in Raschka's rasbt/LLMs-from-scratch repository (103.8k stars), whose bonus chapters implement Llama 3.2, Qwen3 dense and MoE, Gemma 3 and 4, Olmo 3 and others — so you can go from 'this one uses MLA' to running code in two clicks. Alongside the alphabetically organised gallery there is an Architecture Diff tool that puts any two models side by side, a linked table of contents covering all 98 entries, and a changelog with an RSS feed so new additions can be tracked. Multimodal components beyond the text decoder are deliberately out of scope. It reached the Hacker News front page in March 2026 at 586 points.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~1-2 hours to browse; ongoing reference thereafter
- Provider
- Sebastian Raschka
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Compare 98 LLM architectures on identical axes instead of reading 98 papers
- ✓Tell dense, sparse-MoE, hybrid and recurrent decoder designs apart at a glance
- ✓Read attention variants such as MHA, GQA and MLA and what each costs at inference
- ✓Estimate KV-cache memory per token when sizing serving hardware and context limits
- ✓Trace a design choice from a gallery card to its config.json and technical report
- ✓Diff any two models side by side to see exactly which components differ
- ✓Connect an architecture to a runnable from-scratch PyTorch implementation
Highlights
- •98 architectures normalised onto identical fields — the tedious comparison work is already done for you
- •Architecture Diff tool for direct two-model side-by-side comparison, which no individual paper gives you
- •KV-cache-per-token figures make it useful for serving and capacity decisions, not only for study
- •Maintained with a changelog and RSS feed, entries through mid-2026, rather than a one-off blog post
- •Cross-linked to the author's rasbt/LLMs-from-scratch repo (103.8k stars) for working implementations
Who It’s For
Best For
- ✓Engineers choosing an open-weight model who need to know its inference profile
- ✓Anyone studying transformer variants for model-design work or interviews
- ✓Serving and infrastructure engineers sizing KV cache and GPU memory budgets
Prerequisites
- •Solid understanding of transformer internals — attention, feed-forward blocks, normalisation
- •Familiarity with terms like MoE, GQA and KV cache; this is a reference, not an introduction
FAQ
What is LLM Architecture Gallery?
A maintained side-by-side reference for 98 text LLM architectures by Sebastian Raschka, author of Build a Large Language Model (From Scratch). Each model gets a card with an architecture diagram, decoder type, attention variant, parameter count, context window, license and KV-cache cost per token, plus links to its config.json, technical report and a from-scratch implementation. Use it to pick an open-weight model, or to see exactly what changed between two releases.
Is LLM Architecture Gallery free?
LLM Architecture Gallery is free to access.
What level is LLM Architecture Gallery for?
LLM Architecture Gallery is aimed at a advanced audience. Recommended background: Solid understanding of transformer internals — attention, feed-forward blocks, normalisation, Familiarity with terms like MoE, GQA and KV cache; this is a reference, not an introduction.
How long does LLM Architecture Gallery take?
Expect roughly ~1-2 hours to browse; ongoing reference thereafter. Most learners work through it at their own pace.
What will I learn from LLM Architecture Gallery?
You'll learn: Compare 98 LLM architectures on identical axes instead of reading 98 papers; Tell dense, sparse-MoE, hybrid and recurrent decoder designs apart at a glance; Read attention variants such as MHA, GQA and MLA and what each costs at inference; Estimate KV-cache memory per token when sizing serving hardware and context limits; Trace a design choice from a gallery card to its config.json and technical report; Diff any two models side by side to see exactly which components differ; Connect an architecture to a runnable from-scratch PyTorch implementation.
Topics
Sources
This page was written from 2 sources, 1 on domains other than sebastianraschka.com.