Opik — Open-Source LLM Evaluation, Tracing and Optimization
by Comet
Trace it, score it, gate it in CI — the whole LLM evaluation loop, self-hostable under Apache-2.0.
Overview
Opik is Comet's Apache-2.0 platform for tracing, evaluating and monitoring LLM applications, RAG systems and agentic workflows, and the repository plus documentation cover four connected areas. Tracing records every LLM call, tool invocation and agent step through an SDK decorator, an OpenTelemetry endpoint, or one of 60+ framework integrations including OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, AutoGen, Google ADK and Flowise, with per-trace cost tracking. Evaluation covers datasets, experiments and test suites written as natural-language assertions, scored by more than 30 pre-built metrics — hallucination, moderation, answer relevance, context precision, plus heuristic and code-based custom metrics — with annotation queues for human review; the documentation's stated philosophy is that test suites should grow organically from real production failures rather than from a separate test-writing phase. A PyTest integration runs those suites in CI so a prompt or model change is gated on evidence instead of vibes. Production monitoring adds dashboards and online evaluation rules that score incoming traces while tracking feedback scores, latency, cost and error rate, and Opik Guardrails covers runtime safety checks. The Opik Agent Optimizer SDK closes the loop by optimising prompts against scored data. It deploys on Comet's managed cloud, locally via Docker Compose, or on Kubernetes with Helm, with no commercial licence required to self-host; the README was last updated 17 July 2026 at version 1.7.0, and the project has 21.4k GitHub stars.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Freemium
- Duration
- ~2 hours to instrument an app and run a first evaluation; ongoing reference thereafter
- Provider
- Comet
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Instrument any LLM application with the SDK decorator or one of 60+ framework integrations
- ✓Trace multi-step agent runs including tool calls, retries and nested spans with cost attribution
- ✓Score outputs with LLM-as-a-judge metrics for hallucination, moderation, relevance and context precision
- ✓Build datasets and experiments so a prompt change is measured rather than guessed at
- ✓Write test suites as natural-language assertions and run them in PyTest on every commit
- ✓Set online evaluation rules that score production traces and surface latency, cost and error trends
- ✓Optimise prompts automatically with the Opik Agent Optimizer SDK against scored data
- ✓Self-host the full platform with Docker Compose or Kubernetes and Helm instead of shipping traces to a vendor
Highlights
- •Apache-2.0 with the entire platform self-hostable — evaluation and monitoring included, not a crippled open core
- •Covers the whole loop in one tool: tracing, offline evals, CI gating and production monitoring
- •The PyTest integration turns evals into ordinary CI, which is the step most eval tooling leaves to you
- •Agent Optimizer closes the loop by rewriting prompts from scored production data rather than intuition
- •21.4k GitHub stars with OpenTelemetry support, so it drops into an existing observability stack
- •Show HN launch drew 86 points and 15 comments, and the project has shipped steadily to v1.7.0 since
Who It’s For
Best For
- ✓AI engineers who ship LLM features and need to prove a change improved something
- ✓Teams with compliance or data-residency constraints that rule out a hosted tracing vendor
- ✓Developers debugging multi-step agent or RAG pipelines that fail non-deterministically
- ✓Platform teams adding evaluation gates to CI before an LLM change reaches production
Prerequisites
- •Python or TypeScript, and an existing LLM application worth instrumenting
- •Familiarity with LLM-as-a-judge scoring and its failure modes
- •Docker or Kubernetes basics if you intend to self-host rather than use the managed cloud
FAQ
What is Opik — Open-Source LLM Evaluation, Tracing and Optimization?
Opik is Comet's open-source platform for tracing, evaluating and monitoring LLM applications, and its repository and documentation together form a complete guide to instrumenting one. It is aimed at AI engineers who ship LLM or agent features and need evidence that a prompt, model or retrieval change actually helped. After working through it you can trace an agent end to end, build datasets and LLM-as-a-judge evaluations, gate them in PyTest CI, and self-host the whole stack on Docker or Kubernetes.
Is Opik — Open-Source LLM Evaluation, Tracing and Optimization free?
Opik — Open-Source LLM Evaluation, Tracing and Optimization offers free content, with paid options for certificates or premium features.
What level is Opik — Open-Source LLM Evaluation, Tracing and Optimization for?
Opik — Open-Source LLM Evaluation, Tracing and Optimization is aimed at a intermediate audience. Recommended background: Python or TypeScript, and an existing LLM application worth instrumenting, Familiarity with LLM-as-a-judge scoring and its failure modes, Docker or Kubernetes basics if you intend to self-host rather than use the managed cloud.
How long does Opik — Open-Source LLM Evaluation, Tracing and Optimization take?
Expect roughly ~2 hours to instrument an app and run a first evaluation; ongoing reference thereafter. Most learners work through it at their own pace.
What will I learn from Opik — Open-Source LLM Evaluation, Tracing and Optimization?
You'll learn: Instrument any LLM application with the SDK decorator or one of 60+ framework integrations; Trace multi-step agent runs including tool calls, retries and nested spans with cost attribution; Score outputs with LLM-as-a-judge metrics for hallucination, moderation, relevance and context precision; Build datasets and experiments so a prompt change is measured rather than guessed at; Write test suites as natural-language assertions and run them in PyTest on every commit; Set online evaluation rules that score production traces and surface latency, cost and error trends; Optimise prompts automatically with the Opik Agent Optimizer SDK against scored data; Self-host the full platform with Docker Compose or Kubernetes and Helm instead of shipping traces to a vendor.
Topics
Sources
This page was written from 4 sources, 3 on domains other than github.com.