MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026)
by Stanford University
Stanford's graduate course on the math of cheaper LLMs: MoE, KV-cache compression, quantization, speculative decoding and efficient post-training.
Overview
MS&E 319 is a Stanford graduate course taught in Autumn 2026 by Amin Karbasi (VP and Chief AI Scientist at Cisco and adjunct professor at Stanford), Anay Mehrotra (Motwani Postdoctoral Fellow at Stanford), Amin Saberi (Professor of Management Science and Engineering at Stanford) and Grigoris Velegkas (Research Scientist at Google Research). It asks how to choose a training objective, model architecture and inference algorithm when compute is limited. The 16 lectures fall into five parts: pre-training, model architecture, efficient inference, post-training, and risks and vulnerabilities. Lecture 1 is an overview and Lecture 2 reviews LLM architecture and training. Lectures 3 to 10 cover multi-token prediction and the Muon optimizer, mixtures of experts, KV caching, sparse attention and compact KV-cache representations, positional embeddings and recurrent memory, vector and matrix quantization, and speculative decoding. Lectures 11 to 16 cover SFT and LoRA, RLHF and DPO, distillation and recursive self-improvement, and the risks of generative AI. Readings for Lectures 2 to 9 pair older papers (Vaswani et al., the Kaplan and Hoffmann scaling-law papers, Shazeer et al. and Fedus et al. on MoE) with recent reports including DeepSeek-V2, V3.2 and V4 and Kimi Linear. As of October 27, 2026 the public site has slides for Lectures 1 to 10, many in static and animated versions, and none yet for Lectures 11 to 16. Grading is two written assignments, one programming assignment on efficiency methods and a closed-book midterm worth 70%, so expect proofs and derivations more than code.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- 10-week term, 16 lectures of 80 minutes each; self-study from the posted slides and readings
- Provider
- Stanford University
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How multi-token prediction and the Muon optimizer make pre-training compute go further
- ✓How mixture-of-experts layers trade total parameters for a smaller active compute budget
- ✓How KV caching, sparse attention and compact KV-cache representations cut inference memory
- ✓How positional embeddings and recurrent memory affect long-context efficiency in transformers
- ✓How vector and matrix quantization methods reduce the memory cost of model weights
- ✓How speculative decoding and related techniques make LLM inference faster and cheaper
- ✓Where SFT, LoRA, RLHF, DPO and distillation fit in an efficient post-training budget
- ✓What risks and vulnerabilities generative models carry, covered in two closing lectures
Highlights
- •Taught by Stanford faculty together with researchers from Cisco and Google Research
- •Readings pair the original transformer, scaling-law and MoE papers with DeepSeek-V3.2, DeepSeek-V4 and Kimi Linear
- •Many lecture decks are posted in both static and animated versions
- •Assignment 1 solutions are public, so self-learners can check their own work
- •Karbasi's announcement on X drew 518 likes and 603 saves, per a Digg snapshot
Who It’s For
Best For
- ✓ML engineers who serve or fine-tune LLMs and need to reason about cost
- ✓Researchers moving into efficient architectures and inference methods
- ✓Graduate students who have finished CS 224N, CS 324 or equivalent coursework
Prerequisites
- •Deep learning and gradient-based optimization at the level of CS 224N, CS 324 or equivalent
- •Algorithm analysis and comfort with basic proofs (CS 161 or CS 261 maturity helps)
FAQ
What is MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026)?
A Stanford graduate course from Autumn 2026 on making large language models cheaper to train, serve and fine-tune. It is for engineers and researchers who already know transformers and want the math behind mixture-of-experts, KV-cache compression, quantization and speculative decoding, so they can reason about efficiency trade-offs in their own models and serving stacks.
Is MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026) free?
MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026) is free to access.
What level is MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026) for?
MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026) is aimed at a advanced audience. Recommended background: Deep learning and gradient-based optimization at the level of CS 224N, CS 324 or equivalent, Algorithm analysis and comfort with basic proofs (CS 161 or CS 261 maturity helps).
How long does MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026) take?
Expect roughly 10-week term, 16 lectures of 80 minutes each; self-study from the posted slides and readings. Most learners work through it at their own pace.
What will I learn from MS&E 319: Efficient Generative Language Models (Stanford, Autumn 2026)?
You'll learn: How multi-token prediction and the Muon optimizer make pre-training compute go further; How mixture-of-experts layers trade total parameters for a smaller active compute budget; How KV caching, sparse attention and compact KV-cache representations cut inference memory; How positional embeddings and recurrent memory affect long-context efficiency in transformers; How vector and matrix quantization methods reduce the memory cost of model weights; How speculative decoding and related techniques make LLM inference faster and cheaper; Where SFT, LoRA, RLHF, DPO and distillation fit in an efficient post-training budget; What risks and vulnerabilities generative models carry, covered in two closing lectures.
Topics
Sources
This page was written from 2 sources, 1 on domains other than web.stanford.edu.