MLModelsFrameworks

15-442/15-642: Machine Learning Systems (CMU, Spring 2026)

by Carnegie Mellon University (Tianqi Chen, Zhihao Jia)

AdvancedCourseFree23 lectures of 80 minutes, now on YouTube; csdiy.wiki estimates 100+ hours including the three programming labs

CMU's ML systems course from Tianqi Chen and Zhihao Jia: autodiff, CUDA, ZeRO, PagedAttention, speculative decoding and Blackwell GEMM kernels.

Start LearningAdded Nov 18, 2026 · Updated Nov 18, 2026

Overview

15-442/15-642 Machine Learning Systems is taught at Carnegie Mellon by Tianqi Chen and Zhihao Jia, with nine teaching assistants. The Spring 2026 offering ran 23 lectures from January to April. It starts with why ML systems matter, deep learning programming abstractions, automatic differentiation and optimizing linear algebra, then moves to GPU architecture and CUDA programming and a transformer and attention case study. The middle third covers ML parallelization (data parallelism and ZeRO, then model and pipeline parallelism), memory optimizations such as tensor rematerialization and offload, and ML compilers covering data layouts and GEMM on modern GPUs. The LLM section covers serving with continuous batching, PagedAttention and RadixAttention, the Blackwell GPU and the TIRx DSL, speculative decoding, parameter-efficient fine-tuning and mixture of experts. It closes with ML compilation, superoptimization, mega-kernels and guest lectures on Google TPUs and LLM agents. Three labs carry the course: an autodiff framework for computational graphs; ZeRO stage 3 sharding and tensor model parallelism in MPI and NumPy; and an FP16 GEMM kernel for the B200 in TIRx, which csdiy.wiki reports going from 0.02 to over 1,300 TFLOP/s, comparable to cuBLAS. CMU published the lectures on YouTube on November 17, 2026 as part of its free CS Learn program.

At a Glance

Topic
ML
Level
Advanced
Format
Course
Cost
Free
Duration
23 lectures of 80 minutes, now on YouTube; csdiy.wiki estimates 100+ hours including the three programming labs
Provider
Carnegie Mellon University (Tianqi Chen, Zhihao Jia)
Hands-on
Yes — code/exercises
Certificate
None

What You’ll Learn

  • ✓Implement an automatic differentiation framework for computational graphs from scratch in Python
  • ✓Program GPUs in CUDA and understand how GPU architecture shapes deep learning performance
  • ✓Apply data parallelism, ZeRO sharding, tensor parallelism and pipeline parallelism to model training
  • ✓Reduce training memory with tensor rematerialization and offloading to heterogeneous compute
  • ✓Explain how LLM serving engines use continuous batching, PagedAttention, RadixAttention and speculative decoding
  • ✓Write a high-performance FP16 GEMM kernel for Blackwell B200 GPUs using the TIRx DSL
  • ✓Understand how ML compilers handle data layouts, superoptimization and mega-kernel generation
  • ✓Reason about the systems cost of mixture-of-experts models and parameter-efficient fine-tuning

Highlights

  • •Taught by CMU faculty Tianqi Chen and Zhihao Jia, who split the lectures between them
  • •Spring 2026 lectures released free on YouTube via CMU CS Learn on November 17, 2026
  • •Covers current LLM serving techniques such as PagedAttention, RadixAttention and speculative decoding
  • •The GEMM lab targets Blackwell B200 features like TMA loads, warp specialization and 2-CTA clusters
  • •CMU CS Learn launch drew 2,305 points and 314 comments on Hacker News (checked November 18, 2026)

Who It’s For

Best For

  • ✓ML engineers who want to understand what vLLM, SGLang and PyTorch do under the hood
  • ✓Engineers moving into GPU kernel, compiler or inference infrastructure roles
  • ✓Graduate students considering ML systems research
  • ✓Platform teams that size and tune distributed training or LLM serving clusters

Prerequisites

  • •An introductory computer systems course such as CMU 15-213
  • •An introductory deep learning course
  • •Comfortable Python plus enough C++ to read CUDA code
  • •Basic linear algebra and calculus

FAQ

What is 15-442/15-642: Machine Learning Systems (CMU, Spring 2026)?

15-442/642 is Carnegie Mellon's Machine Learning Systems course, taught by Tianqi Chen and Zhihao Jia, with the Spring 2026 lectures released free on YouTube in November 2026. It is for engineers who want to understand how frameworks, distributed training and LLM serving engines work underneath, and to write their own autodiff, parallelism and GPU kernels.

Is 15-442/15-642: Machine Learning Systems (CMU, Spring 2026) free?

15-442/15-642: Machine Learning Systems (CMU, Spring 2026) is free to access.

What level is 15-442/15-642: Machine Learning Systems (CMU, Spring 2026) for?

15-442/15-642: Machine Learning Systems (CMU, Spring 2026) is aimed at a advanced audience. Recommended background: An introductory computer systems course such as CMU 15-213, An introductory deep learning course, Comfortable Python plus enough C++ to read CUDA code, Basic linear algebra and calculus.

How long does 15-442/15-642: Machine Learning Systems (CMU, Spring 2026) take?

Expect roughly 23 lectures of 80 minutes, now on YouTube; csdiy.wiki estimates 100+ hours including the three programming labs. Most learners work through it at their own pace.

What will I learn from 15-442/15-642: Machine Learning Systems (CMU, Spring 2026)?

You'll learn: Implement an automatic differentiation framework for computational graphs from scratch in Python; Program GPUs in CUDA and understand how GPU architecture shapes deep learning performance; Apply data parallelism, ZeRO sharding, tensor parallelism and pipeline parallelism to model training; Reduce training memory with tensor rematerialization and offloading to heterogeneous compute; Explain how LLM serving engines use continuous batching, PagedAttention, RadixAttention and speculative decoding; Write a high-performance FP16 GEMM kernel for Blackwell B200 GPUs using the TIRx DSL; Understand how ML compilers handle data layouts, superoptimization and mega-kernel generation; Reason about the systems cost of mixture-of-experts models and parameter-efficient fine-tuning.

Topics

ml systemscudadistributed trainingllm servinggpu kernelscmu

Sources

This page was written from 5 sources, 3 on domains other than mlsyscourse.org.

  1. 1.mlsyscourse.org — mlsyscourse.orgvendor
  2. 2.mlsyscourse.org — schedulevendor
  3. 3.cs.cmu.edu — learn
  4. 4.csdiy.wiki — CMU15 442
  5. 5.hn.algolia.com — 50522980