DeepEval Documentation — The LLM Evaluation Framework
by Confident AI
Unit-test your LLM app like code: 50+ ready-made metrics, pytest-native assertions, and traces you can run in CI.
Overview
DeepEval is an open-source LLM evaluation framework for LLM applications, documented at deepeval.com. It ships 50+ ready-to-use metrics spanning LLM-as-a-judge, agent, tool-use, conversational, safety, RAG and multimodal evaluation, so most common failure modes have an off-the-shelf scorer before you write a custom one. Evaluations are expressed as unit tests: the framework is built for pytest compatibility and integrates with CI providers, which makes it practical to block a regression on a prompt or retrieval change the same way you would block a failing build. Alongside metrics, the docs cover datasets (collections that make evaluation repeatable across prompts and models), end-to-end versus component-level evals, and tracing — runtime records of agent steps, spans and component behavior — for debugging multi-step agents. DeepEval is local-first: evaluations run in your own environment with no account required for core functionality, and you only need an LLM provider key (for example OPENAI_API_KEY) for metrics that use an LLM judge. Syncing runs to Confident AI's hosted platform for shared dashboards and production observability is optional. The docs are community-shaped and note integration with AI coding tools including Cursor, Claude Code and Codex.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced reference; ~5-minute quickstart
- Provider
- Confident AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Write LLM and agent evals as pytest-style unit tests and wire them into CI/CD
- ✓Choose among 50+ built-in metrics — LLM-as-a-judge, RAG faithfulness and relevancy, tool-use, conversational, safety, multimodal
- ✓Build reusable evaluation datasets so results are comparable across prompts and models
- ✓Run end-to-end evals versus component-level evals on individual pipeline steps
- ✓Trace agent runs by spans and steps to find which component actually failed
- ✓Keep evaluation local-first, adding a hosted dashboard only if the team needs it
Highlights
- •pytest-native — evals live next to your test suite instead of in a notebook
- •50+ research-backed metrics out of the box, including agent and tool-use metrics
- •Local-first and open-source; no account needed for the core framework
- •Covers both RAG and multi-step agent evaluation, plus tracing for debugging
Who It’s For
Best For
- ✓AI engineers adding automated quality gates to an LLM or RAG app
- ✓Teams replacing manual spot-checking with regression tests on prompts
- ✓Agent developers who need tool-use and trajectory-level metrics
- ✓Platform engineers wiring LLM evals into an existing CI pipeline
Prerequisites
- •Working Python and pytest experience
- •An LLM application or agent you already run
- •An LLM provider API key for judge-based metrics
FAQ
What is DeepEval Documentation — The LLM Evaluation Framework?
The official documentation for DeepEval, an open-source, local-first evaluation framework that lets you write LLM and agent evals as pytest-style unit tests. Aimed at AI engineers who need repeatable, automated quality gates for RAG pipelines, agents and prompt changes instead of manual spot-checking.
Is DeepEval Documentation — The LLM Evaluation Framework free?
DeepEval Documentation — The LLM Evaluation Framework is free to access.
What level is DeepEval Documentation — The LLM Evaluation Framework for?
DeepEval Documentation — The LLM Evaluation Framework is aimed at a intermediate audience. Recommended background: Working Python and pytest experience, An LLM application or agent you already run, An LLM provider API key for judge-based metrics.
How long does DeepEval Documentation — The LLM Evaluation Framework take?
Expect roughly Self-paced reference; ~5-minute quickstart. Most learners work through it at their own pace.
What will I learn from DeepEval Documentation — The LLM Evaluation Framework?
You'll learn: Write LLM and agent evals as pytest-style unit tests and wire them into CI/CD; Choose among 50+ built-in metrics — LLM-as-a-judge, RAG faithfulness and relevancy, tool-use, conversational, safety, multimodal; Build reusable evaluation datasets so results are comparable across prompts and models; Run end-to-end evals versus component-level evals on individual pipeline steps; Trace agent runs by spans and steps to find which component actually failed; Keep evaluation local-first, adding a hosted dashboard only if the team needs it.