nanochat: The best ChatGPT that $100 can buy
by Andrej Karpathy
Train a ChatGPT-style model end to end, from Rust tokenizer to web UI, for about $100.
Overview
nanochat is Andrej Karpathy's minimal, dependency-light codebase that trains a ChatGPT-style model end to end on a single 8xH100 node, and it is the successor in spirit to nanoGPT. A single speedrun.sh runs the whole pipeline: a custom Rust BPE tokenizer (rustbpe) with a 65,536-token vocabulary reaching 4.8x compression on FineWeb, pretraining on 11.2B tokens of FineWeb-EDU across 21,400 iterations (~3 hours), midtraining on a 568K-example mixture of SmolTalk, MMLU auxiliary and GSM8K to teach conversation format, multiple choice and tool use (~8 minutes), supervised finetuning on padded conversations matching the inference format (~7 minutes), and optional reinforcement learning on GSM8K with objective rewards (~1.5 hours). Inference ships as a custom Engine class with KV caching and prefill/decode separation, exposed through both scripts.chat_cli and a FastAPI web interface at scripts.chat_web. Every run emits a report card scoring CORE, ARC-Easy, ARC-Challenge, MMLU, GSM8K, HumanEval and ChatCORE, so you can see exactly what $100 buys — the walkthrough's run took 3h51m and $92.40 through SFT at $24/hour on Lambda. Model scale is set by one --depth dial; depth 26 approaches GPT-2 XL capability. MIT licensed, 57,229 stars, and a public time-to-GPT-2 leaderboard tracks reproductions.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Tutorial
- Cost
- Free
- Duration
- ~4 hours of GPU time for the speedrun (~$92 on an 8xH100 node); a weekend to read the code
- Provider
- Andrej Karpathy
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How the full LLM pipeline fits together from tokenizer to served chat endpoint
- ✓Training a BPE tokenizer in Rust and measuring its compression ratio against GPT-2 and GPT-4
- ✓Running pretraining on FineWeb-EDU shards across an eight-GPU H100 node
- ✓Why midtraining exists and what a SmolTalk, MMLU and GSM8K mixture teaches
- ✓Applying supervised finetuning and reinforcement learning with objective GSM8K rewards
- ✓Implementing efficient inference with KV caching and prefill/decode separation
- ✓Reading a model report card across CORE, ARC, MMLU, GSM8K and HumanEval
Highlights
- •One script, one node, one --depth dial — the entire pipeline stays legible in a weekend
- •Karpathy's companion walkthrough (discussion #1) annotates every stage with real timings and costs
- •Honest about the ceiling: the $100 model is roughly kindergarten-grade and the report card proves it
- •57,229 stars, MIT licensed and still actively pushed, with community ports to JAX and TPUs
- •A public time-to-GPT-2 leaderboard turns the repo into a reproducible benchmark, not just a demo
Who It’s For
Best For
- ✓Engineers who want to understand LLM training by reading all of it
- ✓Practitioners moving from fine-tuning hosted APIs to training models from scratch
- ✓Researchers needing a small, hackable baseline for pretraining experiments
Prerequisites
- •Solid PyTorch and transformer fundamentals — this is not an introduction
- •Access to a multi-GPU node (8xH100 recommended) and roughly $100 of compute budget
- •Comfort with the Linux command line and long-running training jobs
FAQ
What is nanochat: The best ChatGPT that $100 can buy?
Andrej Karpathy's minimal, hackable codebase that trains a full ChatGPT-style model from scratch on a single 8xH100 node: tokenizer, pretraining, midtraining, supervised finetuning, optional reinforcement learning, inference and a chat UI. For engineers who want to understand the whole LLM pipeline by reading and running all of it rather than calling an API.
Is nanochat: The best ChatGPT that $100 can buy free?
nanochat: The best ChatGPT that $100 can buy is free to access.
What level is nanochat: The best ChatGPT that $100 can buy for?
nanochat: The best ChatGPT that $100 can buy is aimed at a advanced audience. Recommended background: Solid PyTorch and transformer fundamentals — this is not an introduction, Access to a multi-GPU node (8xH100 recommended) and roughly $100 of compute budget, Comfort with the Linux command line and long-running training jobs.
How long does nanochat: The best ChatGPT that $100 can buy take?
Expect roughly ~4 hours of GPU time for the speedrun (~$92 on an 8xH100 node); a weekend to read the code. Most learners work through it at their own pace.
What will I learn from nanochat: The best ChatGPT that $100 can buy?
You'll learn: How the full LLM pipeline fits together from tokenizer to served chat endpoint; Training a BPE tokenizer in Rust and measuring its compression ratio against GPT-2 and GPT-4; Running pretraining on FineWeb-EDU shards across an eight-GPU H100 node; Why midtraining exists and what a SmolTalk, MMLU and GSM8K mixture teaches; Applying supervised finetuning and reinforcement learning with objective GSM8K rewards; Implementing efficient inference with KV caching and prefill/decode separation; Reading a model report card across CORE, ARC, MMLU, GSM8K and HumanEval.
Topics
Sources
This page was written from 4 sources, 2 on domains other than github.com.