microgpt — Train and Run a GPT in 200 Lines of Pure Python
by Andrej Karpathy
Every piece of a GPT (tokenizer, autograd, attention, Adam, sampling) in one dependency-free 200-line Python file you can read in a sitting
Overview
microgpt is a guided walkthrough, published by Andrej Karpathy on 12 February 2026, of a single file of about 200 lines of pure Python with no dependencies (no PyTorch, NumPy or CUDA) that trains a GPT and then samples from it. Karpathy calls it the culmination of micrograd, makemore, nanoGPT and a decade-long effort to simplify LLMs to their bare essentials, and says he cannot simplify it any further. The post follows the file from top to bottom. Dataset: roughly 32,000 names, one per document. Tokenizer: 27 IDs, the 26 lowercase letters plus a BOS token. Autograd: a scalar Value class that records the computation graph and backpropagates with the chain rule. Parameters: 4,192 weights, with an embedding width of 16, 4 attention heads, 1 layer and a block size of 16. Architecture: a GPT-2-style transformer using RMSNorm, multi-head attention with an explicit KV cache, an MLP and residual connections. Training loop: cross-entropy loss and Adam at a 0.01 learning rate with linear decay. Inference: temperature sampling starting from BOS. Run as-is, 1,000 steps take about a minute on a MacBook and bring the loss from about 3.3 to 2.37, producing plausible new names. Later sections show the build-up in stages, from train0.py (a bigram count table with no neural net) to train5.py (Adam), and list what production systems do differently: BPE tokenizers, trillions of training tokens, SFT and RL post-training, and batched, paged and speculative inference. The post closes with an FAQ and links a Google Colab notebook and the source gist.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Tutorial
- Cost
- Free
- Duration
- ~1-2 hours to read and step through; the script trains in about 1 minute on a laptop
- Provider
- Andrej Karpathy
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How a character-level tokenizer with a BOS token maps names to 27 integer IDs
- ✓How to build a scalar autograd engine (the Value class) that backpropagates through the chain rule
- ✓How token and position embeddings, multi-head attention, an MLP, RMSNorm and residual connections compose a GPT-2-style model
- ✓How an explicit KV cache works when tokens are processed one at a time, and why it stays in the graph during training
- ✓How cross-entropy loss, the Adam optimizer and linear learning-rate decay drive a complete training loop
- ✓How temperature-controlled sampling starting from the BOS token generates new text at inference time
- ✓How the design evolves stage by stage from a bigram count table (train0.py) to the full GPT with Adam (train5.py)
- ✓What separates this toy model from production LLMs: BPE tokenizers, data scale, SFT and RL post-training, and batched speculative inference
Highlights
- •The entire pipeline (dataset, tokenizer, autograd, model, optimizer, training and inference) fits in about 200 lines with zero dependencies: no PyTorch, no NumPy, no CUDA
- •Runs in about a minute on a laptop (1,000 steps, loss from ~3.3 to ~2.37), with a ready-made Google Colab notebook linked from the post
- •A staged progression from train0.py to train5.py lets you see each concept being added, one at a time
- •A dedicated 'real stuff' section maps every simplification to what production systems do instead, so the toy does not mislead
- •Drew 1,936 points and 324 comments on Hacker News, where practitioners posted ports to C++, Rust, Zig, Go and TypeScript and browser visualizers of the model
Who It’s For
Best For
- ✓Application engineers who call LLM APIs daily and want to understand what happens inside the model
- ✓Engineers preparing to read nanoGPT, nanochat or a production transformer codebase
- ✓Educators and team leads looking for a one-sitting explanation of GPT internals to share with a team
Prerequisites
- •Comfortable reading Python, including classes and operator overloading
- •Basic calculus (derivatives and the chain rule) to follow the autograd section
- •Helpful but optional: prior exposure to neural networks, backpropagation or Karpathy's earlier micrograd
FAQ
What is microgpt — Train and Run a GPT in 200 Lines of Pure Python?
microgpt is Andrej Karpathy's February 2026 walkthrough of a single 200-line, dependency-free Python file that trains a GPT and samples from it. It is for developers who use LLMs every day but have never seen the tokenizer, autograd, attention, optimizer and sampling loop together in one place. Afterwards you can read, modify and explain a transformer training loop line by line.
Is microgpt — Train and Run a GPT in 200 Lines of Pure Python free?
microgpt — Train and Run a GPT in 200 Lines of Pure Python is free to access.
What level is microgpt — Train and Run a GPT in 200 Lines of Pure Python for?
microgpt — Train and Run a GPT in 200 Lines of Pure Python is aimed at a intermediate audience. Recommended background: Comfortable reading Python, including classes and operator overloading, Basic calculus (derivatives and the chain rule) to follow the autograd section, Helpful but optional: prior exposure to neural networks, backpropagation or Karpathy's earlier micrograd.
How long does microgpt — Train and Run a GPT in 200 Lines of Pure Python take?
Expect roughly ~1-2 hours to read and step through; the script trains in about 1 minute on a laptop. Most learners work through it at their own pace.
What will I learn from microgpt — Train and Run a GPT in 200 Lines of Pure Python?
You'll learn: How a character-level tokenizer with a BOS token maps names to 27 integer IDs; How to build a scalar autograd engine (the Value class) that backpropagates through the chain rule; How token and position embeddings, multi-head attention, an MLP, RMSNorm and residual connections compose a GPT-2-style model; How an explicit KV cache works when tokens are processed one at a time, and why it stays in the graph during training; How cross-entropy loss, the Adam optimizer and linear learning-rate decay drive a complete training loop; How temperature-controlled sampling starting from the BOS token generates new text at inference time; How the design evolves stage by stage from a bigram count table (train0.py) to the full GPT with Adam (train5.py); What separates this toy model from production LLMs: BPE tokenizers, data scale, SFT and RL post-training, and batched speculative inference.
Topics
Sources
This page was written from 3 sources, 2 on domains other than karpathy.github.io.