CS336: Language Modeling from Scratch
by Stanford University
Write the tokenizer, the transformer, the kernels and the RL loop yourself - nothing is imported for you.
Overview
CS336 is taught by Tatsunori Hashimoto and Percy Liang, and its premise is that you cannot understand language models by calling them. Across five units - the heaviest implementation load in Stanford's AI curriculum, with deliberately minimal scaffolding - students build the stack themselves. The lecture sequence runs tokenization, PyTorch and resource accounting, architectures and hyperparameters, mixture of experts, GPUs, Triton kernels, two lectures on parallelism, scaling laws, inference, evaluation, two on data, and three on alignment covering supervised fine-tuning and reinforcement learning, plus guest lectures. Five assignments carry the real work: Basics, where you write the tokenizer, the transformer and the optimizer and train a small language model; Systems, where you profile attention, implement FlashAttention-2 and build distributed training; Scaling, where you fit scaling laws; Data, where you filter and deduplicate Common Crawl; and Alignment and Reasoning RL, which is supervised fine-tuning followed by RL for mathematical reasoning. Everything is public: handouts and starter code live in the stanford-cs336 GitHub organisation, where assignment1-basics alone carries around 2,500 stars and the lectures repository about 3,600, and lecture videos are posted to YouTube - the course page reached 558 points on Hacker News, which is where most self-study cohorts find it. The 2026 offering is current and the 2025 materials remain online. Self-study is feasible but neither cheap nor quick: you need real GPU time, which the course site addresses by pointing at rented H100s.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- Full 5-unit university course; ~19 lectures and 5 large assignments, months of self-study
- Provider
- Stanford University
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Implement a byte-pair-encoding tokenizer, a transformer and an optimizer from scratch
- ✓Train a small language model end to end and account for its compute and memory cost
- ✓Profile attention and implement FlashAttention-2 for measurable speedups
- ✓Write custom GPU kernels in Triton and reason about memory bandwidth limits
- ✓Build data-parallel and tensor-parallel training across multiple devices
- ✓Fit scaling laws and use them to pick model size against data budget
- ✓Filter and deduplicate Common Crawl into a usable pretraining corpus
- ✓Run supervised fine-tuning and reinforcement learning for math reasoning
Highlights
- •You build the model - there is no course-provided transformer to import and call
- •Assignment handouts, starter code, slides and lecture videos are all public and free
- •The starter repositories are heavily used: roughly 2,500 stars on assignment 1 alone
- •Reached 558 points on Hacker News, which is how most self-study cohorts find it
- •Covers the systems half - kernels, parallelism, inference - that nearly every other LLM course skips
Who It’s For
Best For
- ✓Engineers targeting LLM pretraining, inference or training-systems roles
- ✓Practitioners who already fine-tune models and want the layer beneath
- ✓Self-studiers with GPU budget and several months of evenings to spend
Prerequisites
- •Strong Python and fluent PyTorch - you write the model, not glue code
- •Prior deep learning plus some systems and performance-optimization experience
- •Linear algebra, calculus, probability and statistics at college level
- •Access to real GPUs; the course points self-studiers at rented H100s
FAQ
What is CS336: Language Modeling from Scratch?
Stanford's implementation-heavy course on building a language model across the whole stack: tokenization, architecture, GPU kernels, parallelism, scaling laws, data curation, inference, evaluation and alignment. Lecture videos, slides, assignment handouts and starter code are all public, so it is genuinely self-studiable if you have real GPU budget and the prerequisites, which are hard and not negotiable.
Is CS336: Language Modeling from Scratch free?
CS336: Language Modeling from Scratch is free to access.
What level is CS336: Language Modeling from Scratch for?
CS336: Language Modeling from Scratch is aimed at a advanced audience. Recommended background: Strong Python and fluent PyTorch - you write the model, not glue code, Prior deep learning plus some systems and performance-optimization experience, Linear algebra, calculus, probability and statistics at college level, Access to real GPUs; the course points self-studiers at rented H100s.
How long does CS336: Language Modeling from Scratch take?
Expect roughly Full 5-unit university course; ~19 lectures and 5 large assignments, months of self-study. Most learners work through it at their own pace.
What will I learn from CS336: Language Modeling from Scratch?
You'll learn: Implement a byte-pair-encoding tokenizer, a transformer and an optimizer from scratch; Train a small language model end to end and account for its compute and memory cost; Profile attention and implement FlashAttention-2 for measurable speedups; Write custom GPU kernels in Triton and reason about memory bandwidth limits; Build data-parallel and tensor-parallel training across multiple devices; Fit scaling laws and use them to pick model size against data budget; Filter and deduplicate Common Crawl into a usable pretraining corpus; Run supervised fine-tuning and reinforcement learning for math reasoning.
Topics
Sources
This page was written from 4 sources, 2 on domains other than cs336.stanford.edu.