An Illustrated Guide to AI Agents
by O'Reilly Media (Maarten Grootendorst and Jay Alammar)
An illustrated walk through the AI agent stack, built up in code around a small TinyAgent.
Overview
An Illustrated Guide to AI Agents: Concepts and Code for Building Agents with LLMs, Tools, and Memory is a 450-page O'Reilly book, with an ebook release date of September 2, 2026. Part I, The Anatomy of an AI Agent, has seven chapters. They cover LLM internals (tokens, training with GRPO, self-attention, KV caching, mixture of experts), reasoning models (test-time compute, verifiers, DeepSeek-R1), memory (agentic RAG and context engineering), tools (function calling, tool learning, the Model Context Protocol and SKILL.md skills), planning and reflection (ReAct, Reflexion), and evaluation (benchmarks, LLM-as-a-judge, trajectory evaluation, pass@k reliability, safety). Part II, Specialized Agents, covers multi-agent systems, multimodal understanding, and code agents, including repository maps, compaction, LLM cache optimization and SWE-bench. A small agent called TinyAgent recurs through the chapters as new abilities are added. The free companion GitHub repository holds one notebook per chapter, which can run in Google Colab on a free T4 GPU or locally; the code is free, but the book text is sold separately. Jay Alammar is a director and engineering fellow at Cohere who writes a popular AI/ML blog. Maarten Grootendorst is a clinical data scientist who maintains the open-source packages BERTopic, KeyBERT and PolyFuzz. Publisher blurbs on the repository come from Chip Huyen, Omar Khattab and Ofir Press. Reader reviews had not appeared when this entry was checked on November 5, 2026.
At a Glance
- Topic
- Agentic
- Level
- Intermediate
- Format
- Book
- Cost
- Paid
- Duration
- 450 pages, 10 chapters, with a companion notebook for each chapter; self-paced
- Provider
- O'Reilly Media (Maarten Grootendorst and Jay Alammar)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Explain how tokens, self-attention, KV caching and mixture-of-experts layers work inside modern LLMs
- ✓Compare how reasoning models use test-time compute, verifiers and reinforcement learning such as DeepSeek-R1
- ✓Manage short-term and long-term agent memory with trimming, summarization, agentic RAG and context engineering
- ✓Define, select and call tools, and connect agents through the Model Context Protocol and SKILL.md skills
- ✓Implement planning and reflection patterns including ReAct, Self-Refine and Reflexion in an agent
- ✓Evaluate agents with public benchmarks, LLM-as-a-judge rubrics, trajectory evaluation and pass@k reliability metrics
- ✓Build code agents using sandboxed interpreters, repository maps, compaction and LLM cache optimization
- ✓Choose orchestration patterns and communication protocols for multi-agent systems and deep research agents
Highlights
- •Covers both how to build agents and the model internals beneath them, from self-attention and GRPO to tools and evals, in one book
- •Current for a print book: MCP, SKILL.md skills, context engineering and pass^k reliability all have their own sections
- •The companion repository had 403 stars and 116 forks on November 5, 2026, with notebooks that run on a free Colab T4 GPU
- •Illustration-led explanations from the authors, with TinyAgent code that grows from chapter to chapter
- •Too new for independent reader reports: its one Hacker News submission (November 2, 2026) had 2 points when checked
Who It’s For
Best For
- ✓Engineers who use agent frameworks and want to understand what happens beneath them
- ✓AI engineers who learn better from diagrams than from a stack of survey papers
- ✓Technical leads and students looking for one structured overview of the agent stack
Prerequisites
- •Working Python to run and modify the chapter notebooks
- •A Google account for Colab's free T4 GPU, or a local Python setup using the repository's quick-start guide
- •Some API keys may be needed for parts of the code; the book's preface has a section on API keys and hardware requirements
FAQ
What is An Illustrated Guide to AI Agents?
An Illustrated Guide to AI Agents is a 450-page O'Reilly book by Maarten Grootendorst and Jay Alammar for engineers and technical readers who want to understand how agents work. It explains LLMs, memory, tools, planning and evaluation through diagrams, and you extend a small TinyAgent in code along the way.
Is An Illustrated Guide to AI Agents free?
An Illustrated Guide to AI Agents is a paid resource.
What level is An Illustrated Guide to AI Agents for?
An Illustrated Guide to AI Agents is aimed at a intermediate audience. Recommended background: Working Python to run and modify the chapter notebooks, A Google account for Colab's free T4 GPU, or a local Python setup using the repository's quick-start guide, Some API keys may be needed for parts of the code; the book's preface has a section on API keys and hardware requirements.
How long does An Illustrated Guide to AI Agents take?
Expect roughly 450 pages, 10 chapters, with a companion notebook for each chapter; self-paced. Most learners work through it at their own pace.
What will I learn from An Illustrated Guide to AI Agents?
You'll learn: Explain how tokens, self-attention, KV caching and mixture-of-experts layers work inside modern LLMs; Compare how reasoning models use test-time compute, verifiers and reinforcement learning such as DeepSeek-R1; Manage short-term and long-term agent memory with trimming, summarization, agentic RAG and context engineering; Define, select and call tools, and connect agents through the Model Context Protocol and SKILL.md skills; Implement planning and reflection patterns including ReAct, Self-Refine and Reflexion in an agent; Evaluate agents with public benchmarks, LLM-as-a-judge rubrics, trajectory evaluation and pass@k reliability metrics; Build code agents using sandboxed interpreters, repository maps, compaction and LLM cache optimization; Choose orchestration patterns and communication protocols for multi-agent systems and deep research agents.
Topics
Sources
This page was written from 4 sources, 3 on domains other than github.com.