GPU MODE Lectures
by GPU MODE
The community lecture series that teaches you to write and profile the GPU kernels behind modern LLMs.
Overview
GPU MODE (formerly CUDA MODE) is a community reading group whose weekly talks have grown into a 106-lecture curriculum, with slides and runnable code for 35 of the lectures published in the Apache-2.0 gpu-mode/lectures repository (6.4k stars). The sequence starts practical rather than theoretical: lecture 1 profiles a custom CUDA kernel inside a PyTorch program using load_inline, Triton and NVIDIA Nsight Compute; lectures 2-4 work through parallel programming and the compute/memory architecture largely following the PMPP textbook; lecture 5 optimizes matrix multiplication with shared memory and tiling; lecture 8 is a consolidated performance checklist covering memory coalescing, occupancy, control divergence, tiling and privatization. From there it becomes a tour of the techniques inside real inference and training stacks: kernel fusion for PyTorch optimizers, GPU quantization and low-bit training, sparsity, parallel reduction, Flash Attention (through FlashAttention 4 in lecture 80) and Ring Attention for long context, Triton, CUTLASS and CuTe layout algebra, NCCL collectives, tensor cores, SASS and GPU microarchitecture, speculative decoding, Metal kernels for Apple GPUs, and Hugging Face kernels in lecture 106. Speakers include Mark Saroufim, Jeremy Howard, Thomas Viehmann, Andreas Koepf, Jane Xu and Charles Hernandez — practitioners shipping the code they present. Lectures are recorded and posted as videos, so the repo functions as the slide-and-code index rather than the whole course.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- 106 lectures, ~1-1.5h each, self-paced; the first 15 form a ~20h CUDA foundation track
- Provider
- GPU MODE
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Profile a custom CUDA kernel inside a PyTorch program with Nsight Compute
- ✓Write CUDA C kernels covering thread organization, memory hierarchy and heterogeneous execution
- ✓Optimize matrix multiplication using shared memory, tiling and coalesced memory access
- ✓Apply a systematic performance checklist: occupancy, control divergence, privatization
- ✓Fuse kernels to accelerate PyTorch optimizers and reduce memory round-trips
- ✓Implement and reason about Flash Attention and Ring Attention for long contexts
- ✓Write Triton kernels and use CUTLASS/CuTe layout algebra for tensor operations
- ✓Understand GPU quantization, low-bit training and sparsity as performance techniques
- ✓Read SASS assembly and connect it back to GPU microarchitecture behaviour
- ✓Use NCCL collective communication for multi-GPU training and inference
Highlights
- •106 lectures with slides and runnable code for 35 of them in one Apache-2.0 repo
- •Taught by PyTorch, Triton and CUDA engineers presenting code they actually ship
- •Covers the frontier, not just fundamentals — FlashAttention 4, CuTe, SASS, NVFP4
- •Sequenced so lecture 1 is profiling a real kernel, not a hello-world grid launch
- •Backed by an active Discord, kernel competitions and working groups, so the series keeps extending
- •Includes Apple Metal kernel lectures, rare in an otherwise CUDA-dominated curriculum
Who It’s For
Best For
- ✓ML engineers whose training or inference is GPU-bound and who need to fix it themselves
- ✓Inference engineers customizing vLLM, SGLang or PyTorch with hand-written kernels
- ✓Performance engineers moving from framework-level tuning into kernel-level work
- ✓Anyone working through the PMPP textbook who wants lectures alongside it
Prerequisites
- •Comfortable Python and PyTorch, including writing custom modules and reading profiles
- •Basic C/C++ — the CUDA lectures assume you can read and build C code
- •Access to a CUDA-capable GPU (or a Colab/cloud instance) to run the lecture code
- •Undergraduate-level computer architecture helps for the memory-hierarchy and SASS lectures
FAQ
What is GPU MODE Lectures?
A free, 106-lecture community series on GPU performance engineering for machine learning, taught by working PyTorch, Triton and CUDA engineers. It is aimed at ML engineers who have hit a wall on inference or training throughput and need to read a profiler, write a custom kernel, and understand why the fast path is fast. After working through it you can profile a PyTorch model in Nsight Compute, write and fuse your own CUDA or Triton kernels, and reason about Flash Attention, quantization and collective communication at the hardware level.
Is GPU MODE Lectures free?
GPU MODE Lectures is free to access.
What level is GPU MODE Lectures for?
GPU MODE Lectures is aimed at a advanced audience. Recommended background: Comfortable Python and PyTorch, including writing custom modules and reading profiles, Basic C/C++ — the CUDA lectures assume you can read and build C code, Access to a CUDA-capable GPU (or a Colab/cloud instance) to run the lecture code, Undergraduate-level computer architecture helps for the memory-hierarchy and SASS lectures.
How long does GPU MODE Lectures take?
Expect roughly 106 lectures, ~1-1.5h each, self-paced; the first 15 form a ~20h CUDA foundation track. Most learners work through it at their own pace.
What will I learn from GPU MODE Lectures?
You'll learn: Profile a custom CUDA kernel inside a PyTorch program with Nsight Compute; Write CUDA C kernels covering thread organization, memory hierarchy and heterogeneous execution; Optimize matrix multiplication using shared memory, tiling and coalesced memory access; Apply a systematic performance checklist: occupancy, control divergence, privatization; Fuse kernels to accelerate PyTorch optimizers and reduce memory round-trips; Implement and reason about Flash Attention and Ring Attention for long contexts; Write Triton kernels and use CUTLASS/CuTe layout algebra for tensor operations; Understand GPU quantization, low-bit training and sparsity as performance techniques; Read SASS assembly and connect it back to GPU microarchitecture behaviour; Use NCCL collective communication for multi-GPU training and inference.
Topics
Sources
This page was written from 4 sources, 2 on domains other than github.com.