AI Evals For Engineers & PMs
by Maven — Hamel Husain & Shreya Shankar (Parlance Labs)
Replace vibes-based debugging with a workflow that tells you whether the change helped.
Overview
AI Evals For Engineers & PMs is a live four-week Maven cohort — roughly 3-5 hours a week across 17 sessions grouped into 11 lessons — taught by Hamel Husain, an ML engineer with 25-plus years including Airbnb and GitHub, and Shreya Shankar, a UC Berkeley PhD in data-processing and production-AI evaluation systems who joins Carnegie Mellon as an assistant professor in 2027. The arc runs from building an agent deliberately designed to be evaluable, through generating synthetic queries across categorical dimensions, coding real traces into a taxonomy of failure modes, designing both code-based assertions and LLM-as-judge evaluators, measuring those judges against human labels with true positive rate, true negative rate and confusion matrices, evaluating retrieval with Recall@K and mean reciprocal rank, attaching evaluators to each stage of a decomposed pipeline, and finally wiring the suite into CI/CD regression testing, safety red-teaming and guardrails, and cost-versus-accuracy tuning. Four optional homework assignments build a running Recipe Bot example end to end, with published solutions and walkthroughs; an independent Arize write-up of that homework documents the five-stage pipeline implemented with the open-source Phoenix tracing stack, which is a useful preview of the method for anyone weighing the price. Enrolment includes recordings with lifetime access, 10-plus office hours, a 1,000-plus member Discord, a 150-plus page course reader and a completion certificate. The course page reports more than 4,500 past students from organisations including OpenAI, Google, Meta, Amazon and Microsoft, and a 4.7 out of 5 rating across 897 reviews. Price is $4,200, with a 20%-plus team discount and private cohorts available.
At a Glance
- Topic
- Agentic
- Level
- Intermediate
- Format
- Course
- Cost
- Paid
- Duration
- 4 weeks, 3-5 hours per week, live cohort with lifetime recording access
- Provider
- Maven — Hamel Husain & Shreya Shankar (Parlance Labs)
- Hands-on
- Yes — code/exercises
- Certificate
- Available
What You’ll Learn
- ✓Run error analysis on real traces and build a failure taxonomy
- ✓Generate synthetic test queries that span the dimensions your users actually vary
- ✓Write LLM-as-judge prompts and validate them with TPR, TNR and confusion matrices
- ✓Evaluate retrieval components using Recall@K and mean reciprocal rank
- ✓Attach per-stage evaluators to a decomposed multi-step agent pipeline
- ✓Red-team an AI product for safety issues and design guardrails around them
- ✓Wire evaluation suites into CI/CD so quality regressions fail the build
- ✓Trade cost against accuracy once evaluators give you a reliable signal
Highlights
- •Teaches error analysis first — the step most eval tooling skips straight past
- •Judges are themselves measured against human labels rather than trusted on faith
- •Four homework assignments build one system, Recipe Bot, end to end with published solutions
- •Lifetime access to recordings, the Discord community and every future cohort
- •Tool-agnostic method; an independent Arize walkthrough implements it with open-source Phoenix
Who It’s For
Best For
- ✓Engineers shipping LLM features who cannot tell whether a change helped
- ✓Product managers who need to specify and read AI quality metrics
- ✓Teams standing up their first evaluation and regression-testing practice
- ✓Anyone moving from prompt tinkering to measurable, repeatable iteration
Prerequisites
- •Basic familiarity with LLM applications; a deep ML background is not required
- •Python for the optional coding homework, which PMs can skip
- •Access to a real or realistic AI product to analyse during the cohort
FAQ
What is AI Evals For Engineers & PMs?
A four-week live cohort course from Hamel Husain and Shreya Shankar teaching a systematic workflow for evaluating and improving LLM applications: error analysis on real traces, failure taxonomies, synthetic test data, LLM-as-judge design and validation, retrieval metrics, and CI regression gates. Aimed at engineers and product managers already shipping AI features who want a repeatable process rather than intuition. Listed at $4,200 with lifetime access to materials and future cohorts.
Is AI Evals For Engineers & PMs free?
AI Evals For Engineers & PMs is a paid resource.
What level is AI Evals For Engineers & PMs for?
AI Evals For Engineers & PMs is aimed at a intermediate audience. Recommended background: Basic familiarity with LLM applications; a deep ML background is not required, Python for the optional coding homework, which PMs can skip, Access to a real or realistic AI product to analyse during the cohort.
How long does AI Evals For Engineers & PMs take?
Expect roughly 4 weeks, 3-5 hours per week, live cohort with lifetime recording access. Most learners work through it at their own pace.
What will I learn from AI Evals For Engineers & PMs?
You'll learn: Run error analysis on real traces and build a failure taxonomy; Generate synthetic test queries that span the dimensions your users actually vary; Write LLM-as-judge prompts and validate them with TPR, TNR and confusion matrices; Evaluate retrieval components using Recall@K and mean reciprocal rank; Attach per-stage evaluators to a decomposed multi-step agent pipeline; Red-team an AI product for safety issues and design guardrails around them; Wire evaluation suites into CI/CD so quality regressions fail the build; Trade cost against accuracy once evaluators give you a reliable signal.
Topics
Sources
This page was written from 2 sources, 1 on domains other than maven.com.