GPU Performance Engineering Resources: A Curriculum
by Wafer AI
A structured path from your first CUDA kernel to the inference and multi-GPU tricks frontier labs actually run.
Overview
Maintained by Emilio at Wafer AI, this open-source curriculum solves the discovery problem in GPU performance engineering: the good material exists but is scattered across papers, blog posts, vendor docs, and codebases. It organizes that material into nine progressive sections — fundamentals of GPU programming and architecture; matrix multiplication through cuBLAS internals; tensor cores and mixed precision up to Blackwell-specific techniques; attention and memory-bound kernels including FlashAttention, PagedAttention, and KV-cache optimization; compiler and DSL approaches spanning Triton, CUTLASS, and CuTe; profiling and optimization with NVIDIA's tooling; AMD and alternative hardware including ROCm, CDNA, and TPU programming; production inference systems covering vLLM, SGLang, TensorRT-LLM, continuous batching, and speculative decoding; and distributed multi-GPU work on communication primitives, parallelism strategies, and kernel fusion. The reading list stays current, citing recent Blackwell architecture analysis and recent research on LLM-generated kernels, and it is explicit about what frontier labs do differently from textbook CUDA. Free under the MIT license.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- 9 sections, self-paced (multi-week)
- Provider
- Wafer AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓GPU architecture fundamentals and how to write and profile your first kernels
- ✓Optimize matmul, tensor-core, and mixed-precision paths including Blackwell-era techniques
- ✓Implement and reason about attention kernels: FlashAttention, PagedAttention, KV-cache layout
- ✓Use Triton, CUTLASS, and CuTe as higher-level alternatives to hand-written CUDA C++
- ✓Profile with NVIDIA tooling to find real bottlenecks instead of guessing
- ✓Understand production inference systems (vLLM, SGLang, TensorRT-LLM, continuous batching, speculative decoding) and multi-GPU parallelism
Highlights
- •Curated and sequenced rather than a flat awesome-list dump
- •Covers AMD ROCm/CDNA and TPUs, not just NVIDIA
- •Bridges kernel writing to real serving stacks (vLLM, SGLang, TensorRT-LLM)
- •MIT licensed, free, and kept current with recent architecture and research material
Who It’s For
Best For
- ✓AI engineers moving from framework-level code to kernel-level optimization
- ✓Inference engineers cutting latency and cost on serving stacks
- ✓ML systems engineers scaling training across multiple GPUs
Prerequisites
- •Strong Python and some C++
- •Working knowledge of deep learning and PyTorch
- •Comfort reading papers and vendor architecture docs
FAQ
What is GPU Performance Engineering Resources: A Curriculum?
A free, MIT-licensed curriculum that sequences the best public material on GPU kernel programming and performance optimization, from fundamentals to frontier-lab practice. It is for AI engineers who need to make training and inference actually fast rather than just call a framework.
Is GPU Performance Engineering Resources: A Curriculum free?
GPU Performance Engineering Resources: A Curriculum is free to access.
What level is GPU Performance Engineering Resources: A Curriculum for?
GPU Performance Engineering Resources: A Curriculum is aimed at a advanced audience. Recommended background: Strong Python and some C++, Working knowledge of deep learning and PyTorch, Comfort reading papers and vendor architecture docs.
How long does GPU Performance Engineering Resources: A Curriculum take?
Expect roughly 9 sections, self-paced (multi-week). Most learners work through it at their own pace.
What will I learn from GPU Performance Engineering Resources: A Curriculum?
You'll learn: GPU architecture fundamentals and how to write and profile your first kernels; Optimize matmul, tensor-core, and mixed-precision paths including Blackwell-era techniques; Implement and reason about attention kernels: FlashAttention, PagedAttention, KV-cache layout; Use Triton, CUTLASS, and CuTe as higher-level alternatives to hand-written CUDA C++; Profile with NVIDIA tooling to find real bottlenecks instead of guessing; Understand production inference systems (vLLM, SGLang, TensorRT-LLM, continuous batching, speculative decoding) and multi-GPU parallelism.