DeepEval Documentation — The LLM Evaluation Framework
by Confident AI
Pytest for LLM apps — run evals as a test suite in CI instead of eyeballing outputs.
Overview
DeepEval describes itself as "Pytest but specialized for unit testing LLM apps", and the framing is literal: you write test cases, assert against metrics, and run them with deepeval test run inside an existing Pytest suite — or Vitest, for the TypeScript port. The metric catalogue is the reason to reach for it over hand-rolled scoring. Custom metrics come from G-Eval, an LLM-as-judge metric you define with plain-language criteria, and DAG, a deterministic decision-graph metric for cases where a judge is too loose. RAG metrics cover answer relevancy, faithfulness, contextual precision, contextual recall and RAGAS. Agentic metrics cover task completion, tool correctness, goal accuracy, step efficiency and plan adherence. Multi-turn metrics cover knowledge retention, conversation completeness and turn relevancy, and there are multimodal, hallucination, bias, toxicity and JSON-correctness metrics besides. Tracing via an @observe decorator captures full agent trajectories, which enables component-level evaluation — scoring an individual retriever, tool call or generation span inside a traced run rather than only the end output. Native integrations exist for LangChain, LangGraph, OpenAI, Anthropic, CrewAI and Pydantic AI. A synthesizer generates test datasets and simulates conversations, standard benchmarks including MMLU, HellaSwag, DROP, GSM8K and BigBenchHard run in roughly ten lines, and red teaming is handled by the sibling DeepTeam project. Confident AI is the optional cloud backend for regression dashboards and production online evals; local-only evaluation remains fully functional without it. Apache-2.0, roughly 17.8k GitHub stars, installed with pip install -U deepeval.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~30 min quickstart; ~4-6 hours for the full metric, tracing and dataset docs
- Provider
- Confident AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Write LLM evaluations as Pytest test cases that fail a CI build
- ✓Define custom judge criteria with G-Eval and deterministic rubrics with DAG
- ✓Score RAG pipelines on faithfulness, contextual precision and contextual recall
- ✓Evaluate agent runs on task completion, tool correctness and plan adherence
- ✓Trace agents with @observe and score individual retriever or tool spans
- ✓Generate synthetic evaluation datasets and simulated multi-turn conversations
- ✓Run MMLU, HellaSwag, GSM8K and BigBenchHard benchmarks in a few lines
Highlights
- •Evals become ordinary tests, so regressions are caught by CI instead of by users
- •Component-level scoring inside a traced agent run, not just final-answer grading
- •Roughly forty ready-made metrics spanning RAG, agents, multi-turn and safety
- •Fully usable offline — the Confident AI cloud platform is optional, not a gate
- •Apache-2.0 with ~17.8k stars and native hooks into LangChain, LangGraph, CrewAI and Pydantic AI
Who It’s For
Best For
- ✓AI engineers who currently judge output quality by reading samples
- ✓Teams adding quality gates to an LLM or RAG deployment pipeline
- ✓Agent developers who need to know which step in a trajectory failed
Prerequisites
- •Python and working familiarity with Pytest
- •An existing LLM, RAG or agent application to evaluate
- •An LLM API key, since most metrics use a judge model
FAQ
What is DeepEval Documentation — The LLM Evaluation Framework?
Documentation for the open-source evaluation framework that turns LLM output quality into ordinary unit tests. It ships ready-made metrics for RAG, agent trajectories, multi-turn conversations and safety, plugs into Pytest and Vitest so evals run in CI, and traces agent runs so you can score individual retrieval or tool-call spans rather than only the final answer.
Is DeepEval Documentation — The LLM Evaluation Framework free?
DeepEval Documentation — The LLM Evaluation Framework is free to access.
What level is DeepEval Documentation — The LLM Evaluation Framework for?
DeepEval Documentation — The LLM Evaluation Framework is aimed at a intermediate audience. Recommended background: Python and working familiarity with Pytest, An existing LLM, RAG or agent application to evaluate, An LLM API key, since most metrics use a judge model.
How long does DeepEval Documentation — The LLM Evaluation Framework take?
Expect roughly ~30 min quickstart; ~4-6 hours for the full metric, tracing and dataset docs. Most learners work through it at their own pace.
What will I learn from DeepEval Documentation — The LLM Evaluation Framework?
You'll learn: Write LLM evaluations as Pytest test cases that fail a CI build; Define custom judge criteria with G-Eval and deterministic rubrics with DAG; Score RAG pipelines on faithfulness, contextual precision and contextual recall; Evaluate agent runs on task completion, tool correctness and plan adherence; Trace agents with @observe and score individual retriever or tool spans; Generate synthetic evaluation datasets and simulated multi-turn conversations; Run MMLU, HellaSwag, GSM8K and BigBenchHard benchmarks in a few lines.
Topics
Sources
This page was written from 2 sources, 1 on domains other than deepeval.com.