ModelsMLAgentic

Building strong safety evaluations

by Anthropic

IntermediateGuideFree~20 min read, longer if you follow the linked papers

Anthropic's method for designing, grading and reporting LLM safety evals, with sample-size floors and grader checks you can copy.

Start LearningAdded Nov 16, 2026 · Updated Nov 16, 2026

Overview

Building strong safety evaluations is a long-form post on Anthropic's engineering blog, dated November 13, 2026 and written by eight Anthropic authors, including Shayne Longpre and Jeremy Hadfield. It defines a safety eval as a measure of a model's propensity to cause harm, separate from capability evals, and walks through five stages. Safety first covers staffing (methods experts paired with domain experts) and running evals with safeguards off inside sandboxes with no internet and real-time monitoring. Design covers observational versus causal designs, construct validity (refusal rate is often the wrong target, so the post uses an appropriate response rate graded against expert rubrics), paired harmful and benign prompts to catch over-refusal, and a four-step multi-turn simulation recipe in which a separate model plays the user. Run covers contamination, saturation (single-turn harmless rates of 94 to 98% on recent models), grader calibration with atomic binary criteria, a different grader model, position-bias checks and Cohen's kappa against human labels, plus preregistration with floors of 100 items for multi-turn and 1,000 for single-turn evals. Report covers per-category statistics, confidence intervals that can exceed 10 points, configuration logging and responsible disclosure. It points to the open-source Inspect framework and Anthropic's Petri auditing toolkit for implementation.

At a Glance

Topic
Models
Level
Intermediate
Format
Guide
Cost
Free
Duration
~20 min read, longer if you follow the linked papers
Provider
Anthropic
Hands-on
No
Certificate
None

What You’ll Learn

  • ✓How to define the harm construct before collecting data, and why raw refusal rate misleads
  • ✓How to pair harmful and benign prompts in one domain to measure over-refusal
  • ✓How to build multi-turn simulations where a second model plays the user from an expert spec
  • ✓How to calibrate an LLM grader with binary rubric criteria, a different model and position-flip checks
  • ✓How to validate grader output against human labels using chance-corrected agreement such as Cohen's kappa
  • ✓How to preregister metrics and thresholds, with 100-item and 1,000-item sample-size floors
  • ✓How to report per-category results with confidence intervals and document model, prompt and grader versions
  • ✓How to run cyber and web-enabled evals safely inside sandboxes with real-time monitoring

Highlights

  • •Gives specific numbers most eval guides leave out: sample-size floors, saturation levels and interval widths of more than 10 points
  • •Includes worked failures from Anthropic's own models, such as a 34-point API versus claude.ai gap on a multi-turn safety eval
  • •Shows a romance-scam eval where a 98% versus 94% difference was not significant, so no winner was declared
  • •Points to open-source tooling you can run: UK AISI's Inspect and Anthropic's Petri, which has about 1.4k GitHub stars
  • •Applies equally to agent behavior evals, since the grader and preregistration checklists do not depend on the harm domain

Who It’s For

Best For

  • ✓ML engineers building safety, red-team or behavior eval suites for LLM products
  • ✓Eval and trust-and-safety teams deciding whether a model or prompt change is a regression
  • ✓Engineers who use LLM-as-judge graders and need a calibration checklist
  • ✓Researchers who publish eval results and want a reproducible reporting standard

Prerequisites

  • •Familiarity with running LLM evals or LLM-as-judge grading
  • •Basic statistics: confidence intervals, paired comparisons and multiple-comparison correction

FAQ

What is Building strong safety evaluations?

Building strong safety evaluations is an Anthropic engineering guide, published November 13, 2026, on how to measure whether a language model causes harm. It is written for ML engineers and eval teams who run safety or behavior evals, and it gives enough concrete rules to rebuild an internal eval suite: construct definitions, multi-turn simulation, grader calibration, preregistration and reporting with confidence intervals.

Is Building strong safety evaluations free?

Building strong safety evaluations is free to access.

What level is Building strong safety evaluations for?

Building strong safety evaluations is aimed at a intermediate audience. Recommended background: Familiarity with running LLM evals or LLM-as-judge grading, Basic statistics: confidence intervals, paired comparisons and multiple-comparison correction.

How long does Building strong safety evaluations take?

Expect roughly ~20 min read, longer if you follow the linked papers. Most learners work through it at their own pace.

What will I learn from Building strong safety evaluations?

You'll learn: How to define the harm construct before collecting data, and why raw refusal rate misleads; How to pair harmful and benign prompts in one domain to measure over-refusal; How to build multi-turn simulations where a second model plays the user from an expert spec; How to calibrate an LLM grader with binary rubric criteria, a different model and position-flip checks; How to validate grader output against human labels using chance-corrected agreement such as Cohen's kappa; How to preregister metrics and thresholds, with 100-item and 1,000-item sample-size floors; How to report per-category results with confidence intervals and document model, prompt and grader versions; How to run cyber and web-enabled evals safely inside sandboxes with real-time monitoring.

Topics

llm evaluationai safetyllm-as-judgered teamingeval designanthropic

Sources

This page was written from 4 sources, 2 on domains other than anthropic.com.

  1. 1.anthropic.com — building strong safety evaluationsvendor
  2. 2.anthropic.com — engineeringvendor
  3. 3.news.ycombinator.com — item
  4. 4.github.com — petri