Evaluating and Improving Agentic Applications
by DeepLearning.AI and Arize AI
Trace an AI agent, validate your LLM judges, and turn production failures into regression tests.
Overview
Evaluating and Improving Agentic Applications has five modules with 28 video lessons, 6 code labs and 10 graded assignments. The course page, updated November 4, 2026, lists 15h 7m in total; the videos, readings and code labs it itemizes add up to about 7.5 hours, and the page does not break out the remainder. Module 1 covers why agents are hard to evaluate and how traces record model and tool calls, with one lab that traces an agent in plain Python and another in Arize Phoenix. Module 2 turns application requirements into test cases, has you label recurring failures, and builds a reusable benchmark called Northwind. Module 3 builds code evaluators, LLM judges and custom rubrics, and measures judge reliability with precision, recall and Cohen's kappa. Module 4 checks tool calls, step order, instruction following across conversation turns, and whether the task was completed. Module 5 compares agent versions on the same cases, works through a regression caused by a change, and converts production failures into regression tests. Laurie Voss, Arize's Head of Developer Relations and a co-founder of npm, teaches the course, and also ran a two-hour evals workshop at AI Engineer Europe 2026 that applied Phoenix to a financial-analysis agent. Phoenix is Arize's own product, and its repository lists the Elastic License 2.0. Free members can watch the videos. Labs, quizzes and certificates require DeepLearning.AI Pro, listed at $25 a month billed annually or $30 billed monthly.
At a Glance
- Topic
- Agentic
- Level
- Intermediate
- Format
- Course
- Cost
- Freemium
- Duration
- 15h 7m as listed by the provider (the itemized videos, readings and code labs add up to about 7.5 hours)
- Provider
- DeepLearning.AI and Arize AI
- Hands-on
- Yes — code/exercises
- Certificate
- Available
What You’ll Learn
- ✓Read traces of model and tool calls, and instrument an agent in both Python and Phoenix
- ✓Turn application requirements into test cases and label the recurring failures you find in them
- ✓Build a reusable benchmark, using the course's Northwind benchmark as the worked example throughout
- ✓Write code evaluators, LLM judges and custom rubrics, then apply them to the same benchmark
- ✓Validate LLM judges by measuring their reliability with precision, recall and Cohen's kappa
- ✓Evaluate tool calls, trajectories, multi-turn instruction following and whether a task was actually completed
- ✓Compare agent versions on identical cases, assess tradeoffs, and decide where to intervene next
- ✓Convert production failures into regression tests that catch the same problem in later agent versions
Highlights
- •Evaluates the evaluators: a full lesson validates LLM judges, with Cohen's kappa alongside precision and recall
- •Goes past final answers to tool calls, step order, conversation turns and task outcomes
- •The last module covers what to do with results, including choosing where to intervene and refining a change after a regression
- •Labs use Arize Phoenix, which had 12.1k GitHub stars and 1.3k forks on November 5, 2026; the vendor teaching the course also makes this tool
- •Updated November 4, 2026, and no independent learner reviews had appeared when this entry was checked on November 5
Who It’s For
Best For
- ✓Developers who have shipped an LLM agent and cannot yet say whether a change improved it
- ✓AI engineers setting up a first evaluation suite and regression process for agents in production
- ✓Teams that rely on LLM judges and need to measure how far those judges can be trusted
Prerequisites
- •Comfort reading and writing Python, plus basic familiarity with pandas
- •Prior experience calling an LLM API; familiarity with tool calling helps but is not required
- •No prior experience with evaluation, tracing or statistics is needed
FAQ
What is Evaluating and Improving Agentic Applications?
Evaluating and Improving Agentic Applications is a DeepLearning.AI course from Arize AI for developers who already write Python and call LLM APIs. It teaches you to trace an agent with Arize Phoenix, build a benchmark and evaluators you can check, then compare agent versions and turn production failures into regression tests.
Is Evaluating and Improving Agentic Applications free?
Evaluating and Improving Agentic Applications offers free content, with paid options for certificates or premium features.
What level is Evaluating and Improving Agentic Applications for?
Evaluating and Improving Agentic Applications is aimed at a intermediate audience. Recommended background: Comfort reading and writing Python, plus basic familiarity with pandas, Prior experience calling an LLM API; familiarity with tool calling helps but is not required, No prior experience with evaluation, tracing or statistics is needed.
How long does Evaluating and Improving Agentic Applications take?
Expect roughly 15h 7m as listed by the provider (the itemized videos, readings and code labs add up to about 7.5 hours). Most learners work through it at their own pace.
What will I learn from Evaluating and Improving Agentic Applications?
You'll learn: Read traces of model and tool calls, and instrument an agent in both Python and Phoenix; Turn application requirements into test cases and label the recurring failures you find in them; Build a reusable benchmark, using the course's Northwind benchmark as the worked example throughout; Write code evaluators, LLM judges and custom rubrics, then apply them to the same benchmark; Validate LLM judges by measuring their reliability with precision, recall and Cohen's kappa; Evaluate tool calls, trajectories, multi-turn instruction following and whether a task was actually completed; Compare agent versions on identical cases, assess tradeoffs, and decide where to intervene next; Convert production failures into regression tests that catch the same problem in later agent versions.
Topics
Sources
This page was written from 4 sources, 3 on domains other than deeplearning.ai.