The Illustrated GPT-2 (Visualizing Transformer LMs)
by Jay Alammar
Watch a decoder-only model turn a prompt into text, one masked-attention step at a time.
Overview
Jay Alammar published this in August 2019 as the decoder-only sequel to The Illustrated Transformer, and it remains the clearest visual account of how a GPT-style model turns a prompt into text. It runs in three parts. Part one covers what a language model is, how GPT-2 differs from BERT by stacking transformer decoder blocks instead of encoders, byte pair encoding tokenization, positional encoding, and a guided tour of the model internals from the token embedding matrix up through the stacked blocks to the output projection; it uses the 124M-parameter small model with its 768-dimension embeddings as the running example and contrasts it against the larger variants. Part two is the heart of the piece: it rebuilds self-attention step by step - creating query, key and value vectors, scoring, and the weighted sum - then shows precisely what masking changes and why generation can reuse earlier work rather than recomputing the whole sequence per token. Part three surveys what the same decoder stack does beyond language modelling, covering machine translation, summarisation, transfer learning and the Music Transformer. There is no runnable code; it points instead at OpenAI's GPT-2 release and the Hugging Face implementation. Read it knowing the model is dated - nothing here on RLHF, mixture of experts or modern context lengths - but the architecture it draws is the one today's chat models still run on, which is why it keeps getting reposted and hit 213 points on Hacker News.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- ~40 minute read (roughly 6,000-7,000 words, diagram-heavy)
- Provider
- Jay Alammar
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How a decoder-only transformer differs from an encoder model such as BERT
- ✓Why masked self-attention is required for autoregressive text generation
- ✓How query, key and value vectors combine into an attention-weighted output
- ✓How byte pair encoding turns raw text into the tokens a model consumes
- ✓How a prompt flows through stacked decoder blocks to produce one next token
- ✓Why caching earlier keys and values makes token-by-token generation practical
- ✓Where the same decoder stack is reused for translation, summarisation and music
Highlights
- •The diagram sequence for masked self-attention is the clearest treatment available anywhere
- •Reached 213 points on Hacker News, and Alammar's illustrated series is the standard reference newcomers get pointed at
- •Bridges the gap between hand-wavy explainers and reading the original papers
- •Free, no signup, no environment to set up - one browser tab and forty minutes
- •Honest limitation: written in 2019 about GPT-2, so nothing on RLHF, MoE or long-context tricks
Who It’s For
Best For
- ✓Engineers who can call an LLM API but cannot picture what happens inside it
- ✓Anyone about to read Attention Is All You Need who wants a map first
- ✓Interview preparation where you have to explain attention out loud
Prerequisites
- •The Illustrated Transformer or an equivalent introduction to attention
- •Comfort with vectors and matrix multiplication - no calculus required
FAQ
What is The Illustrated GPT-2 (Visualizing Transformer LMs)?
Jay Alammar's three-part illustrated walkthrough of GPT-2 and decoder-only language models. It builds up from what a language model is, through byte pair encoding and the transformer decoder block, into a step-by-step reconstruction of masked self-attention and autoregressive generation, then surveys other uses of the same stack. It is the clearest visual account of what actually happens between your prompt and the next token.
Is The Illustrated GPT-2 (Visualizing Transformer LMs) free?
The Illustrated GPT-2 (Visualizing Transformer LMs) is free to access.
What level is The Illustrated GPT-2 (Visualizing Transformer LMs) for?
The Illustrated GPT-2 (Visualizing Transformer LMs) is aimed at a intermediate audience. Recommended background: The Illustrated Transformer or an equivalent introduction to attention, Comfort with vectors and matrix multiplication - no calculus required.
How long does The Illustrated GPT-2 (Visualizing Transformer LMs) take?
Expect roughly ~40 minute read (roughly 6,000-7,000 words, diagram-heavy). Most learners work through it at their own pace.
What will I learn from The Illustrated GPT-2 (Visualizing Transformer LMs)?
You'll learn: How a decoder-only transformer differs from an encoder model such as BERT; Why masked self-attention is required for autoregressive text generation; How query, key and value vectors combine into an attention-weighted output; How byte pair encoding turns raw text into the tokens a model consumes; How a prompt flows through stacked decoder blocks to produce one next token; Why caching earlier keys and values makes token-by-token generation practical; Where the same decoder stack is reused for translation, summarisation and music.
Topics
Sources
This page was written from 2 sources, 1 on domains other than jalammar.github.io.