GPU Performance Engineering Resources: A Curriculum
by Emilio Andere / Wafer
A tiered reading order through GPU kernel work, from PMPP to what frontier labs actually ship.
Overview
Emilio Andere of Wafer maintains this repository as a sequenced reading list, not a tutorial — the stated method is to read the Tier 1 material for every topic first, then return for Tier 2, so a reader builds breadth before depth. It has roughly 1,300 stars and 164 forks. Fundamentals cover the Programming Massively Parallel Processors textbook, the community-run GPU MODE lecture series, NVIDIA's CUDA Programming Guide, and microbenchmarking write-ups on Hopper and Blackwell down to PTX and SASS. Matrix multiplication gets its own section: tiling, shared memory, the well-known from-scratch SGEMM write-ups, and cuBLAS internals. Tensor cores and mixed precision run from Volta through Blackwell's tcgen05 and TMEM, including the FP8, FP4 and NVFP4 microscaling formats. Attention covers FlashAttention v1 to v3, PagedAttention, and KV-cache variants such as GQA, MQA and MLA. Compiler and DSL approaches cover Triton, CUTLASS and CuTe, TileLang, ThunderKittens, TVM, MLIR and Mojo. Later sections handle Nsight Compute and roofline analysis, occupancy and coalescing, CUDA Graphs and warp specialisation, AMD ROCm and CDNA, TPU programming with JAX Pallas, production inference systems including vLLM, SGLang and TensorRT-LLM, LLM-generated kernels such as KernelBench and AlphaEvolve, and NCCL-based multi-GPU communication. Contributions must be primary sources; surface-level tutorials and unverified AI-generated content are explicitly rejected.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- 100+ linked primary sources arranged in tiers across 11 topic areas — months of self-paced study, not a weekend
- Provider
- Emilio Andere / Wafer
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Follow a tiered reading order that builds breadth across GPU topics before depth in any one
- ✓Understand the CUDA memory hierarchy and how shared-memory tiling makes matrix multiplication fast
- ✓Read tensor core and mixed precision material spanning Volta through Blackwell's tcgen05 and TMEM
- ✓Work through FlashAttention v1 to v3, PagedAttention, and the GQA, MQA and MLA cache variants
- ✓Compare kernel authoring approaches across Triton, CUTLASS, CuTe, TileLang, ThunderKittens and Mojo
- ✓Use Nsight Compute and roofline analysis to decide whether a kernel is compute or memory bound
- ✓Locate the current research on LLM-generated kernels, including KernelBench, KernelLLM and AlphaEvolve
Highlights
- •Tiered ordering solves the real problem with kernel resources — there are hundreds and no obvious sequence
- •Explicit quality bar: primary sources only, with surface-level and AI-generated content rejected by policy
- •Spans the full stack from PTX and SASS up to vLLM, SGLang and TensorRT-LLM serving systems
- •Covers non-NVIDIA hardware too — AMD ROCm and CDNA, plus TPU programming with JAX Pallas
- •Points at the GPU MODE community and its 23,000-member Discord, where the lectures are discussed live
Who It’s For
Best For
- ✓Engineers moving from PyTorch-level work into writing and optimising GPU kernels
- ✓AI infrastructure and inference engineers who need to diagnose kernel-level bottlenecks
- ✓Anyone preparing for a GPU performance engineering role and needing a defensible study plan
Prerequisites
- •Strong C++ and comfort with low-level systems concepts such as memory hierarchies and concurrency
- •Working knowledge of deep learning fundamentals and the transformer architecture
- •Access to an NVIDIA GPU to actually run and profile the material the curriculum links to
FAQ
What is GPU Performance Engineering Resources: A Curriculum?
A curated, tiered curriculum of over 100 primary sources for learning GPU performance engineering with a focus on AI infrastructure. It is a reading order rather than a course: read Tier 1 across every topic, then Tier 2, and so on, moving from CUDA fundamentals through tensor cores, FlashAttention, Triton and CUTLASS, profiling, and production inference systems to LLM-generated kernels and multi-GPU communication.
Is GPU Performance Engineering Resources: A Curriculum free?
GPU Performance Engineering Resources: A Curriculum is free to access.
What level is GPU Performance Engineering Resources: A Curriculum for?
GPU Performance Engineering Resources: A Curriculum is aimed at a advanced audience. Recommended background: Strong C++ and comfort with low-level systems concepts such as memory hierarchies and concurrency, Working knowledge of deep learning fundamentals and the transformer architecture, Access to an NVIDIA GPU to actually run and profile the material the curriculum links to.
How long does GPU Performance Engineering Resources: A Curriculum take?
Expect roughly 100+ linked primary sources arranged in tiers across 11 topic areas — months of self-paced study, not a weekend. Most learners work through it at their own pace.
What will I learn from GPU Performance Engineering Resources: A Curriculum?
You'll learn: Follow a tiered reading order that builds breadth across GPU topics before depth in any one; Understand the CUDA memory hierarchy and how shared-memory tiling makes matrix multiplication fast; Read tensor core and mixed precision material spanning Volta through Blackwell's tcgen05 and TMEM; Work through FlashAttention v1 to v3, PagedAttention, and the GQA, MQA and MLA cache variants; Compare kernel authoring approaches across Triton, CUTLASS, CuTe, TileLang, ThunderKittens and Mojo; Use Nsight Compute and roofline analysis to decide whether a kernel is compute or memory bound; Locate the current research on LLM-generated kernels, including KernelBench, KernelLLM and AlphaEvolve.
Topics
Sources
This page was written from 2 sources, 1 on domains other than github.com.