Arize Phoenix Documentation — AI Observability and Evaluation
by Arize AI
Trace every LLM call, retrieval, and agent step with OpenTelemetry — then score them with LLM-as-judge evals, datasets, and experiments.
Overview
Arize Phoenix is an open-source AI observability and evaluation platform, built by Arize AI and the open-source community on top of OpenTelemetry, that 'helps you understand and improve AI applications by giving you a workflow for debugging and iteration.' The documentation is organized around four pillars. Tracing lets you 'see what happened during a single run of your AI application, step by step' — Phoenix accepts traces over OTLP and ships auto-instrumentation for frameworks including LlamaIndex, LangChain, DSPy, Mastra, and the Vercel AI SDK, for providers including OpenAI, Bedrock, and Anthropic, and across Python, TypeScript, and Java. Evaluations let you 'measure the output quality of your application', scoring traces and spans with LLM-based evaluators, code-based checks, or human labels so failures are tracked consistently instead of anecdotally. Prompt engineering support lets you 'iterate on prompts using real examples from your application' with versioning, span replay, and testing. Datasets and experiments let you 'test changes systematically using the same inputs' — group traces into datasets, rerun them through different versions of your application, and compare evaluation results side by side. Getting started requires no signup, API key, or cloud dependency: pip install arize-phoenix, launch the app, and you have a tracing UI on localhost. Phoenix runs in a notebook, a container, on Docker or Kubernetes, or in the cloud of your choice; the open-source package is free to self-host, while Arize offers paid hosted tiers.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Freemium
- Duration
- Self-paced reference; tracing running locally in ~15 minutes
- Provider
- Arize AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Instrument an LLM, RAG, or agent application with OpenTelemetry/OpenInference and capture end-to-end traces
- ✓Debug a single run step by step — every LLM call, retrieval, and tool invocation in one span tree
- ✓Build LLM-as-judge, code-based, and human-label evaluators to score output quality consistently
- ✓Evaluate RAG-specific failure modes such as retrieval relevance and answer groundedness/hallucination
- ✓Turn production traces into datasets and run repeatable experiments to compare prompt or model versions
- ✓Version and replay prompts against real application examples instead of ad-hoc playground testing
- ✓Self-host Phoenix locally, in Docker, or on Kubernetes with no vendor API key required
Highlights
- •Open source and OpenTelemetry-native — no proprietary agent, no lock-in on your trace data
- •Runs locally with one pip install: no signup, no API key, no cloud dependency for the basics
- •Auto-instrumentation across the frameworks engineers actually use (LangChain, LlamaIndex, DSPy, Mastra, Vercel AI SDK, agent SDKs)
- •Covers the full loop — trace, evaluate, dataset, experiment — rather than tracing alone
Who It’s For
Best For
- ✓AI engineers taking an LLM, RAG, or agent app from demo to production
- ✓Teams that need evals and regression testing before shipping prompt or model changes
- ✓Engineers debugging agent runs that fail in ways logs alone don't explain
- ✓Anyone who needs self-hosted LLM observability for data-residency or cost reasons
Prerequisites
- •Python, TypeScript, or Java experience
- •An existing LLM/RAG/agent application to instrument
- •Basic familiarity with tracing/OpenTelemetry concepts is helpful but not required
FAQ
What is Arize Phoenix Documentation — AI Observability and Evaluation?
Official documentation for Arize Phoenix, the open-source AI observability and evaluation platform for LLM and agent applications. It is for AI engineers who need to see exactly what their agent or RAG pipeline did on a given run, and to measure output quality systematically rather than by eyeballing outputs.
Is Arize Phoenix Documentation — AI Observability and Evaluation free?
Arize Phoenix Documentation — AI Observability and Evaluation offers free content, with paid options for certificates or premium features.
What level is Arize Phoenix Documentation — AI Observability and Evaluation for?
Arize Phoenix Documentation — AI Observability and Evaluation is aimed at a intermediate audience. Recommended background: Python, TypeScript, or Java experience, An existing LLM/RAG/agent application to instrument, Basic familiarity with tracing/OpenTelemetry concepts is helpful but not required.
How long does Arize Phoenix Documentation — AI Observability and Evaluation take?
Expect roughly Self-paced reference; tracing running locally in ~15 minutes. Most learners work through it at their own pace.
What will I learn from Arize Phoenix Documentation — AI Observability and Evaluation?
You'll learn: Instrument an LLM, RAG, or agent application with OpenTelemetry/OpenInference and capture end-to-end traces; Debug a single run step by step — every LLM call, retrieval, and tool invocation in one span tree; Build LLM-as-judge, code-based, and human-label evaluators to score output quality consistently; Evaluate RAG-specific failure modes such as retrieval relevance and answer groundedness/hallucination; Turn production traces into datasets and run repeatable experiments to compare prompt or model versions; Version and replay prompts against real application examples instead of ad-hoc playground testing; Self-host Phoenix locally, in Docker, or on Kubernetes with no vendor API key required.