tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers
by Chi (skyzh) and Connor
Build Qwen3 inference, a mini vLLM and a coding agent from raw MLX array operations on a Mac.
Overview
tiny-llm is an Apache-2.0 course on GitHub (about 5.7k stars and 357 commits on main when checked on November 10, 2026) with a companion book at skyzh.github.io/tiny-llm. It is written by Chi (skyzh), a systems software engineer at Neon, which Databricks acquired, with Connor, a TiKV engineer at PingCAP. The course builds a Qwen3-4B model and its serving path without high-level neural-network layers: you write each operator in Python, C++ or Metal, and MLX acts as the correctness oracle and the performance baseline. Week 1, 'From Matmul to Text', covers attention, RoPE, grouped-query attention, RMSNorm and the MLP, model loading, decoding and sampling. Week 2, 'A Faster Single Request', was rebuilt as a profiling-led route through KV caching, packed 4-bit weights, SIMD matrix prefill, fused primitives and tiled prefill attention. Week 3, 'Build a Mini vLLM', adds continuous batching, chunked prefill, a paged KV cache, paged attention and paged FlashAttention, with optional mixture-of-experts and speculative decoding chapters. Week 4 reuses the stack in a local coding agent with checkpoints, history compaction, operator steering, outcome evaluation and bounded tool output. Every chapter ships student exercises (tiny_llm), a reference solution (tiny_llm_ref) and tests. The repo was still being revised on November 7, 2026, and parts of Weeks 2 and 4 are marked in progress.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Course
- Cost
- Free
- Duration
- 4 weekly modules of 5 to 9 day-sized chapters, self-paced; the authors give no hour estimate
- Provider
- Chi (skyzh) and Connor
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Implement attention, RoPE, grouped-query attention, RMSNorm and the MLP from raw MLX array operations
- ✓Load Qwen3 weights and write the decoding loop and sampling code that turns logits into text
- ✓Add a KV cache and measure how it changes per-token latency for a single request
- ✓Keep 4-bit quantized weights packed and write a quantized matrix-vector kernel for decode
- ✓Build continuous batching and chunked prefill so many requests share one serving loop
- ✓Implement a paged KV cache, direct paged attention and paged FlashAttention like vLLM does
- ✓Optionally add mixture-of-experts routing and speculative decoding to the serving engine
- ✓Build a local coding agent with workspace inspection, validated edits, checkpoints and history compaction
Highlights
- •Each chapter ships starter code, a reference solution and tests, and MLX acts as the correctness oracle
- •Runs on an Apple Silicon Mac, so you need no rented CUDA GPU; the book maps Qwen3 sizes to 8 GB through 64 GB of unified memory
- •About 5.7k GitHub stars, and a 297-point Hacker News launch thread in April 2025; commits were still landing on November 7, 2026
- •Covers the systems layer that most LLM courses skip: paged attention, continuous batching and chunked prefill written by hand
- •A community CUDA port (amagilabs/tiny-llm-cuda) exists for people without a Mac
- •The README states what it leaves out: compressed KV caches, cross-request prefix caching, fine-tuning and long-context techniques
Who It’s For
Best For
- ✓Systems and backend engineers who want to understand what vLLM or SGLang does under the hood
- ✓ML infrastructure engineers tuning inference latency and throughput who want to write the kernels once themselves
- ✓Engineers on Apple Silicon Macs who want serious hands-on LLM work without paying for GPU time
- ✓Agent builders who want to see how a coding agent loop sits on top of a serving stack they wrote
Prerequisites
- •Deep learning basics and familiarity with PyTorch; the book points to CMU's Introduction to Machine Learning and Deep Learning Systems courses
- •Comfortable Python plus enough C++ to read and write kernels, and willingness to learn Metal
- •An Apple Silicon Mac, since the tests compare against MLX and PyTorch or NumPy backends are not supported
- •A working grasp of the transformer architecture: attention, embeddings and the decoding loop
FAQ
What is tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers?
tiny-llm is a free, open-source course for systems engineers who want to know how LLM serving works inside. You implement Qwen3 inference from MLX array operations on an Apple Silicon Mac, then add KV caching, quantized kernels, continuous batching and paged attention, and finish by building a local coding agent on top of your own serving stack.
Is tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers free?
tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers is free to access.
What level is tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers for?
tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers is aimed at a advanced audience. Recommended background: Deep learning basics and familiarity with PyTorch; the book points to CMU's Introduction to Machine Learning and Deep Learning Systems courses, Comfortable Python plus enough C++ to read and write kernels, and willingness to learn Metal, An Apple Silicon Mac, since the tests compare against MLX and PyTorch or NumPy backends are not supported, A working grasp of the transformer architecture: attention, embeddings and the decoding loop.
How long does tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers take?
Expect roughly 4 weekly modules of 5 to 9 day-sized chapters, self-paced; the authors give no hour estimate. Most learners work through it at their own pace.
What will I learn from tiny-llm: LLM Inference Serving on Apple Silicon for Systems Engineers?
You'll learn: Implement attention, RoPE, grouped-query attention, RMSNorm and the MLP from raw MLX array operations; Load Qwen3 weights and write the decoding loop and sampling code that turns logits into text; Add a KV cache and measure how it changes per-token latency for a single request; Keep 4-bit quantized weights packed and write a quantized matrix-vector kernel for decode; Build continuous batching and chunked prefill so many requests share one serving loop; Implement a paged KV cache, direct paged attention and paged FlashAttention like vLLM does; Optionally add mixture-of-experts routing and speculative decoding to the serving engine; Build a local coding agent with workspace inspection, validated edits, checkpoints and history compaction.
Topics
Sources
This page was written from 4 sources, 2 on domains other than github.com.