Evaluating AI Agents
by DeepLearning.AI (with Arize AI)
Build an agent, trace every step, then prove whether it actually works.
Overview
A DeepLearning.AI short course built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and Aman Khan, Director of Product, running 2 hours 36 minutes across 15 video lessons, 6 code examples and one graded assignment. It alternates short concept videos with 9-17 minute notebook labs that implement what was just explained. Lesson 2 establishes why evaluating an LLM system is not traditional software testing; lesson 3 decomposes an agent into router, skills and memory, and Lab 1 has you build that agent from scratch. Lessons 5 and 6 add observability, instrumenting the agent so every step it takes emits a trace you can visualise and debug. Lessons 7 and 8 cover router and skill evaluations, where you deliberately choose between code-based graders, LLM-as-a-judge and human annotation for each component, and build test examples out of the traces you already collected. Lessons 9 and 10 move up a level to trajectory evaluation, judging the path an agent took rather than only its final answer, and lessons 11 and 12 organise individual evals into structured experiments so prompt and parameter changes can be compared rather than eyeballed. The course closes with a lesson on improving your LLM-as-a-judge prompt and one on monitoring agents in production. Tooling is Arize Phoenix, the company's open-source, framework-agnostic tracing and evaluation platform. Basic Python is the only stated prerequisite; the course is free to watch, with the graded assignment and certificate behind a PRO membership.
At a Glance
- Topic
- Agentic
- Level
- Beginner
- Format
- Course
- Cost
- Freemium
- Duration
- 2h 36m, self-paced: 15 video lessons, 6 code examples, 1 graded assignment
- Provider
- DeepLearning.AI (with Arize AI)
- Hands-on
- Yes — code/exercises
- Certificate
- Available
What You’ll Learn
- ✓Decompose an agent into router, skills and memory, then build one from scratch
- ✓Instrument an agent so every step it takes emits a visualisable trace
- ✓Pick between code-based, LLM-as-a-judge and human evaluators per component
- ✓Turn collected traces into concrete test examples for router and skill evals
- ✓Evaluate the trajectory an agent took, not just its final answer
- ✓Group individual evals into experiments that compare prompt and parameter changes
- ✓Calibrate an LLM-as-a-judge prompt so its scores hold up under scrutiny
- ✓Carry the same evaluations into production monitoring after deployment
Highlights
- •Taught by the Arize AI team that builds Phoenix, so the tooling advice is first-hand rather than second-hand
- •Alternating concept-and-lab structure: six runnable notebooks, with labs longer than the lectures
- •Covers trajectory evaluation, which most agent-eval material skips entirely
- •Free to watch; only the graded assignment and certificate require a PRO membership
- •Phoenix is open source and framework-agnostic, so the skills transfer off the course platform
Who It’s For
Best For
- ✓Engineers who have shipped an agent and cannot tell whether it regressed
- ✓Teams adopting tracing and observability for LLM applications
- ✓Developers evaluating Arize Phoenix against LangSmith or Langfuse
Prerequisites
- •Basic Python, since the labs are Jupyter notebooks
- •Familiarity with LLM prompting is recommended but not required
- •An LLM API key to run the notebook examples
FAQ
What is Evaluating AI Agents?
A DeepLearning.AI short course built with Arize AI that has you build an agent from scratch, instrument it with tracing, and evaluate each component with code-based, LLM-as-a-judge and human evaluators. It covers router, skill and trajectory evaluation, structuring evals into experiments, and carrying the same measurements into production monitoring.
Is Evaluating AI Agents free?
Evaluating AI Agents offers free content, with paid options for certificates or premium features.
What level is Evaluating AI Agents for?
Evaluating AI Agents is aimed at a beginner audience. Recommended background: Basic Python, since the labs are Jupyter notebooks, Familiarity with LLM prompting is recommended but not required, An LLM API key to run the notebook examples.
How long does Evaluating AI Agents take?
Expect roughly 2h 36m, self-paced: 15 video lessons, 6 code examples, 1 graded assignment. Most learners work through it at their own pace.
What will I learn from Evaluating AI Agents?
You'll learn: Decompose an agent into router, skills and memory, then build one from scratch; Instrument an agent so every step it takes emits a visualisable trace; Pick between code-based, LLM-as-a-judge and human evaluators per component; Turn collected traces into concrete test examples for router and skill evals; Evaluate the trajectory an agent took, not just its final answer; Group individual evals into experiments that compare prompt and parameter changes; Calibrate an LLM-as-a-judge prompt so its scores hold up under scrutiny; Carry the same evaluations into production monitoring after deployment.
Topics
Sources
This page was written from 3 sources, 2 on domains other than deeplearning.ai.