Product Evals in Three Simple Steps
by Eugene Yan
Eugene Yan's three-step process for labeling data, aligning an LLM judge and running evals on every change
Overview
Product Evals in Three Simple Steps is a guide by Ziyou (Eugene) Yan, published in November 2025 on eugeneyan.com, on building evaluations for an LLM product feature. It has three sections. The first covers labeling: sample real inputs and outputs into a spreadsheet, use binary pass/fail or win/lose/tie labels in place of 1 to 5 scales, and aim for roughly 50 to 100 failures in 200 or more samples, sourcing organic failures from smaller models and using active learning to surface more. The second section covers aligning an LLM evaluator. Split the labels about 75/25 into development and held-out test sets, build one evaluator per dimension instead of a single "God Evaluator," swap control and treatment positions in pairwise judgments and record a tie when the verdict flips, and measure precision, recall and Cohen's kappa with priority on recall for the fail class. Yan sets human annotators as the baseline, noting that human kappa is often around 0.2 to 0.3 and that tired reviewers can miss up to half of defects. The third section describes an eval harness that runs evaluators in parallel within rate limits, writes metrics as a single-row dataframe for spreadsheet review, and plugs into the experiment pipeline so a model migration, such as Haiku 3.5 to Haiku 4.5, becomes a one-line config change. The guide closes with sample-size arithmetic showing why 200 samples cannot confirm a 5% defect target at a 3% observed rate while 400 samples can.
At a Glance
- Topic
- ML
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- ~9 min read
- Provider
- Eugene Yan
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How to sample and label 200 or more production outputs with binary pass/fail labels
- ✓Why 1 to 5 rating scales are hard for both humans and LLM judges to calibrate
- ✓How to source realistic failure cases from smaller models and through active learning
- ✓How to split labels into development and held-out test sets for evaluator prompt iteration
- ✓How to build one LLM evaluator per quality dimension and combine them with simple rules
- ✓How to reduce position bias in pairwise judgments by swapping control and treatment
- ✓How to measure evaluator alignment with precision, recall and Cohen's kappa
- ✓How to choose a sample size that can confirm a defect-rate target at 95% confidence
Highlights
- •Gives concrete numbers: 50 to 100 fail cases, a 75/25 split, kappa thresholds and confidence intervals
- •Benchmarks the LLM evaluator against human annotators, whose kappa is often 0.2 to 0.3
- •Separates guardrail metrics that block a release from northstar metrics tracked over time
- •Outputs results as a single-row dataframe that product managers can paste into Excel and compare across runs
Who It’s For
Best For
- ✓AI engineers setting up their first evals for an LLM feature
- ✓Product managers who need defensible quality metrics for prompt or model changes
- ✓Teams planning a model migration who need regression checks before switching
Prerequisites
- •Experience shipping or prototyping an LLM-powered feature
- •Basic statistics, including precision, recall and confidence intervals
FAQ
What is Product Evals in Three Simple Steps?
A short, practical guide by Eugene Yan on building product evaluations for LLM features. It is for AI engineers and product managers who need to know whether a prompt, retrieval or model change made outputs better or worse. Afterwards you can label a balanced dataset, align an LLM evaluator to human judgment and run an eval harness on each config change.
Is Product Evals in Three Simple Steps free?
Product Evals in Three Simple Steps is free to access.
What level is Product Evals in Three Simple Steps for?
Product Evals in Three Simple Steps is aimed at a intermediate audience. Recommended background: Experience shipping or prototyping an LLM-powered feature, Basic statistics, including precision, recall and confidence intervals.
How long does Product Evals in Three Simple Steps take?
Expect roughly ~9 min read. Most learners work through it at their own pace.
What will I learn from Product Evals in Three Simple Steps?
You'll learn: How to sample and label 200 or more production outputs with binary pass/fail labels; Why 1 to 5 rating scales are hard for both humans and LLM judges to calibrate; How to source realistic failure cases from smaller models and through active learning; How to split labels into development and held-out test sets for evaluator prompt iteration; How to build one LLM evaluator per quality dimension and combine them with simple rules; How to reduce position bias in pairwise judgments by swapping control and treatment; How to measure evaluator alignment with precision, recall and Cohen's kappa; How to choose a sample size that can confirm a defect-rate target at 95% confidence.
Topics
Sources
This page was written from 3 sources, 2 on domains other than eugeneyan.com.