CMU 11-868: Large Language Model Systems (Spring 2026)
by Carnegie Mellon University
Write the CUDA kernels behind FlashAttention, paged attention and MoE, then benchmark your own GPT-2 training run.
Overview
The published syllabus spans weeks 1 to 17: GPU programming basics and GPU acceleration (weeks 1-3); transformers and pretrained LLMs (week 4); tokenization, embedding, generation and speculative decoding (week 5); transformer acceleration on GPU with guest lectures on TPU, JAX and Pallas (weeks 6-7); distributed model training and mixture-of-experts (weeks 9-10); memory optimization, quantization and FlashAttention (weeks 11-12); LLM serving, efficient fine-tuning, paged attention and disaggregated systems (weeks 13-15); and final project presentations in week 17. Seven homework assignments are published on the companion assignments site: CUDA programming; the miniTorch framework with automatic differentiation; transformer architecture; transformer CUDA acceleration; distributed training and parallelism; advanced training and inference systems; and an introduction to RLHF. miniTorch was originally written by Sasha Rush for teaching and is extended here from pure Python to real CUDA C++ kernels bound through a CUDA backend, with students implementing low-level tensor operators themselves. The class meets Mondays and Wednesdays 12:30-1:50pm in Scaife Hall 105; grading is 44% homework, 44% project (proposal, mid-term report, presentation, final report, with teams formed in week 6), 10% quiz and 2% participation. No textbook is required — reading is recent LLM systems papers, with Programming Massively Parallel Processors, 4th edition recommended for GPU background. Lei Li is an associate professor in CMU's Language Technologies Institute working on generative AI, machine translation, agentic LLMs and LLM security.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- One semester (weeks 1-17 in the published syllabus), 7 homeworks plus a team project; ~120 hours total per an independent student review
- Provider
- Carnegie Mellon University
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Write CUDA C++ kernels for tensor operations and bind them into Python
- ✓Extend an automatic-differentiation framework from pure Python to GPU execution
- ✓Implement a decoder-only GPT-2 and train it on IWSLT14 German-English
- ✓Fuse and accelerate transformer operations, including FlashAttention-style attention
- ✓Apply distributed training and parallelism strategies, including mixture-of-experts routing
- ✓Reduce memory footprint through quantization and activation memory optimization
- ✓Reason about serving throughput using paged attention and disaggregated prefill and decode
- ✓Speed up generation with speculative decoding and efficient fine-tuning methods
Highlights
- •You implement the kernels rather than call them — the first assignment is raw CUDA, not a PyTorch wrapper
- •Explicitly covers the current serving stack: paged attention, disaggregated prefill/decode, speculative decoding and MoE
- •Guest lectures on TPU, JAX and Pallas widen it past the NVIDIA-only view most GPU courses take
- •An independent review (csdiy.wiki) rates difficulty 4 out of 5 at roughly 120 hours, and warns the distributed-training assignment is hard to configure for self-study
- •Slides and all seven assignment handouts are public on the course and companion homework sites
Who It’s For
Best For
- ✓Engineers moving from using LLMs to optimizing how they are trained and served
- ✓Inference and training infrastructure engineers who need kernel-level fluency
- ✓Researchers who want to benchmark their own serving or parallelism ideas
- ✓Anyone who has finished a deep learning course and wants the systems half of it
Prerequisites
- •Linear algebra, calculus, and probability and statistics
- •Programming in Python plus C, C++ or Java at roughly the level of CMU 15-122
- •Prior deep learning or advanced NLP coursework is strongly recommended — the independent review says it is not suitable for beginners
- •Access to an NVIDIA GPU; the distributed-training assignment needs multiple GPUs and is the hardest part to reproduce alone
FAQ
What is CMU 11-868: Large Language Model Systems (Spring 2026)?
Carnegie Mellon's Spring 2026 offering of 11-868, taught by Lei Li, on the systems layer underneath large language models: GPU programming, transformer acceleration, distributed training, quantization, memory optimization and serving. Assignments extend the miniTorch teaching framework with real CUDA kernels and build a GPT-2 trained on IWSLT14 German-English. The public site carries the syllabus, lecture slides and every assignment handout, so a self-learner with a GPU can follow it.
Is CMU 11-868: Large Language Model Systems (Spring 2026) free?
CMU 11-868: Large Language Model Systems (Spring 2026) is free to access.
What level is CMU 11-868: Large Language Model Systems (Spring 2026) for?
CMU 11-868: Large Language Model Systems (Spring 2026) is aimed at a advanced audience. Recommended background: Linear algebra, calculus, and probability and statistics, Programming in Python plus C, C++ or Java at roughly the level of CMU 15-122, Prior deep learning or advanced NLP coursework is strongly recommended — the independent review says it is not suitable for beginners, Access to an NVIDIA GPU; the distributed-training assignment needs multiple GPUs and is the hardest part to reproduce alone.
How long does CMU 11-868: Large Language Model Systems (Spring 2026) take?
Expect roughly One semester (weeks 1-17 in the published syllabus), 7 homeworks plus a team project; ~120 hours total per an independent student review. Most learners work through it at their own pace.
What will I learn from CMU 11-868: Large Language Model Systems (Spring 2026)?
You'll learn: Write CUDA C++ kernels for tensor operations and bind them into Python; Extend an automatic-differentiation framework from pure Python to GPU execution; Implement a decoder-only GPT-2 and train it on IWSLT14 German-English; Fuse and accelerate transformer operations, including FlashAttention-style attention; Apply distributed training and parallelism strategies, including mixture-of-experts routing; Reduce memory footprint through quantization and activation memory optimization; Reason about serving throughput using paged attention and disaggregated prefill and decode; Speed up generation with speculative decoding and efficient fine-tuning methods.
Topics
Sources
This page was written from 3 sources, 1 on domains other than llmsystem.github.io.