Neural Networks: Zero to Hero
by Andrej Karpathy
Eight lectures, fourteen hours, one GPT built line by line from a scalar autograd engine.
Overview
Neural Networks: Zero to Hero is Andrej Karpathy's free video series that builds a working language model from nothing, writing and explaining every line. It is eight lectures totalling roughly fourteen and a half hours: "The spelled-out intro to neural networks and backpropagation: building micrograd" (2h25m), "The spelled-out intro to language modeling: building makemore" (1h57m), then makemore Part 2: MLP (1h15m), Part 3: Activations, Gradients and BatchNorm (1h55m), Part 4: Becoming a Backprop Ninja (1h56m) and Part 5: Building a WaveNet (56m), followed by "Let's build GPT: from scratch, in code, spelled out" (1h56m) and "Let's build the GPT Tokenizer" (2h13m). The arc is deliberate: micrograd gives you a scalar autograd engine you understand completely, makemore walks the same character-level task up through bigrams, multilayer perceptrons, batch normalization and a dilated hierarchical architecture, and the GPT lectures implement self-attention and a full transformer block following "Attention is All You Need". The companion repository, karpathy/nn-zero-to-hero, is MIT-licensed with about 23,900 stars and carries Jupyter notebooks for most lectures; exercises are posted in the video descriptions or as Colab notebooks, and there is a Discord channel for learners. Karpathy states the prerequisites plainly: solid Python and introductory mathematics, meaning derivatives and Gaussian distributions. One caveat — the series is billed as ongoing, but the repository's most recent commit is from August 2024.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Video
- Cost
- Free
- Duration
- ~14.5 hours across 8 lectures, self-paced
- Provider
- Andrej Karpathy
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Implement reverse-mode autograd from scratch and understand every gradient
- ✓Build a character-level language model from bigrams up to a multilayer perceptron
- ✓Diagnose activation and gradient statistics, then fix them with batch normalization
- ✓Backpropagate manually through a whole network without relying on autograd
- ✓Construct a dilated, hierarchical WaveNet-style architecture for sequence modelling
- ✓Write self-attention and a full transformer block following the original paper
- ✓Build a byte-pair-encoding tokenizer and see why tokenization causes LLM oddities
Highlights
- •Everything is built from scratch — no framework abstraction hides the mechanism
- •Taught by Andrej Karpathy, who writes micrograd, makemore and the GPT code live on screen
- •MIT-licensed companion repository with about 23,900 stars and notebooks for most lectures
- •Exercises for each lecture, plus a Discord channel for learners
- •Honest caveat: billed as an ongoing series, but the repository's last commit was August 2024
Who It’s For
Best For
- ✓Engineers who want to understand LLM internals at the level of code
- ✓Anyone who can use PyTorch but could not rebuild its core
- ✓Practitioners preparing to read or implement transformer research papers
- ✓Self-learners who prefer building something to watching slides
Prerequisites
- •Solid Python programming, as stated on the course page
- •Introductory mathematics — derivatives and Gaussian distributions
- •Roughly fifteen hours, plus the time to type the code along yourself
FAQ
What is Neural Networks: Zero to Hero?
Andrej Karpathy's eight-lecture, roughly fourteen-and-a-half-hour video series that builds neural networks and a GPT from first principles, typing every line on screen. You start with a scalar autograd engine and finish with a working transformer and a byte-pair-encoding tokenizer, understanding backpropagation and attention at the level of the code rather than the diagram.
Is Neural Networks: Zero to Hero free?
Neural Networks: Zero to Hero is free to access.
What level is Neural Networks: Zero to Hero for?
Neural Networks: Zero to Hero is aimed at a advanced audience. Recommended background: Solid Python programming, as stated on the course page, Introductory mathematics — derivatives and Gaussian distributions, Roughly fifteen hours, plus the time to type the code along yourself.
How long does Neural Networks: Zero to Hero take?
Expect roughly ~14.5 hours across 8 lectures, self-paced. Most learners work through it at their own pace.
What will I learn from Neural Networks: Zero to Hero?
You'll learn: Implement reverse-mode autograd from scratch and understand every gradient; Build a character-level language model from bigrams up to a multilayer perceptron; Diagnose activation and gradient statistics, then fix them with batch normalization; Backpropagate manually through a whole network without relying on autograd; Construct a dilated, hierarchical WaveNet-style architecture for sequence modelling; Write self-attention and a full transformer block following the original paper; Build a byte-pair-encoding tokenizer and see why tokenization causes LLM oddities.
Topics
Sources
This page was written from 2 sources, 1 on domains other than karpathy.ai.