Opik
by Comet
Open-source tracing, evaluation and guardrails for LLM applications and AI agents
Opik is Comet's Apache-2.0 licensed observability and evaluation platform for LLM applications, RAG systems and agentic workflows. It traces every step an agent takes, scores output with LLM-as-a-judge metrics, manages prompts and enforces guardrails in production. It is built for AI engineering teams who need to prove an agent works before shipping it, and who would rather self-host than send traces to a vendor.
Opik is an open-source LLM observability and evaluation platform built by Comet, the New York MLOps vendor founded in 2017 by Gideon Mendels and Nimrod Lahav. Released in mid-2024, it has grown to roughly 21,000 GitHub stars under an Apache-2.0 licence and covers four jobs that teams otherwise stitch together from separate tools. Tracing captures full trace trees for multi-step agents, including tool calls, retrieval steps and conversation logs, with token and cost accounting attached. Evaluation runs LLM-as-a-judge scoring across 30-plus built-in metrics covering hallucination, relevance, moderation and task completion, against versioned datasets, and a PyTest plugin lets those evaluations run as CI gates rather than as a manual review step. Prompt management provides versioning and a playground, and the Agent Optimizer SDK applies six automated optimisation algorithms to improve prompts from production data, which is a capability competing open-source observability tools do not offer. Production monitoring adds dashboards, alerting and built-in guardrails for PII detection, topic blocking and prompt-injection checks. Opik ships 60-plus integrations including OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, AutoGen, Google ADK, LiteLLM and OpenTelemetry. It runs three ways: fully self-hosted via Docker Compose or Kubernetes and Helm with no feature gating, on Comet's managed cloud, or as an enterprise deployment with SSO, service accounts, SLAs and SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR coverage. Comet reports Opik handling over 40 million traces daily across more than 150,000 developers and roughly 450 enterprise, startup and academic teams, and names Netflix, Uber, Etsy, Shopify, NatWest and Ancestry among its users. Comet was recognised in the February 2026 Gartner Market Guide for AI Evaluation and Observability Platforms.
The platform or AI engineering lead who is putting an agent into production and needs traces, evals and guardrails inside their own infrastructure, either because of a data-residency constraint or because per-seat observability pricing will not survive the team scaling.
A full trace-plus-evaluation loop that runs self-hosted with unlimited spans and unlimited seats at zero licence cost, so quality gates can be wired into CI before an agent ships.
At a Glance
- Category
- Developer Tools
- Pricing
- Free, Freemium, Subscription, Usage-based, Contact for pricing
- Target Market
- AI Engineers, Machine Learning Engineers, Enterprise Developers, Data Scientists, Platform Engineering Leads
- Deployment
- Open-source, Self-hosted, Cloud-first, API-based
- Founded
- 2017
- Headquarters
- New York, New York, United States
- Team Size
- 51-200
- Customers
- Comet reports 150,000+ developers and roughly 450 enterprise, startup and academic teams, with over 40 million traces processed daily; named users include Netflix, Uber, Etsy, Shopify, NatWest, Ancestry, AssemblyAI and Zappos
Key Features
- ✓Agent tracing
Captures full trace trees for multi-step agents including tool calls, retrieval and conversation logs, with token and cost accounting attached to every span.
- ✓LLM-as-a-judge evaluation
Scores outputs against 30-plus built-in metrics covering hallucination, relevance, moderation and task completion, run over versioned datasets and experiments.
- ✓PyTest CI/CD plugin
Runs evaluations as part of continuous integration, so a quality regression fails the build instead of surfacing after deployment.
- ✓Agent Optimizer
Six automated prompt optimisation algorithms that improve prompts from production data, a capability competing open-source observability tools do not ship.
- ✓Built-in guardrails
PII detection, topic blocking and prompt-injection protection run inline, so compliance controls do not require a second vendor.
- ✓Prompt management and playground
Version-controlled prompts with a built-in playground for side-by-side experimentation before changes reach the application.
- ✓Full self-hosting
Apache-2.0 licensed with Docker Compose and Kubernetes/Helm deployments carrying the same feature set as the cloud version, with no gating.
Capabilities
Use Cases
- •Diagnosing an agent that silently fails
Replay the full trace of tool calls and retrieved context to find which step returned bad data rather than guessing from the final answer.
- •Shipping RAG changes safely
Score retrieval relevance and hallucination against a fixed dataset on every pull request, so a chunking change cannot quietly degrade answers.
- •Regulated self-hosted deployment
Run the full platform inside the company network so traces containing customer data never reach a third-party observability vendor.
- •Controlling coding-agent spend
Track token and cost data per trace across engineering teams to see which agents and prompts are actually driving the model bill.
- •Production guardrails
Block PII leakage, off-topic responses and prompt-injection attempts inline while alerting on anomalies in live traffic.
Ideal For
Best For
- ✓Debugging multi-step agent failures by replaying the full trace tree of tool calls and retrieval steps
- ✓Running LLM-as-a-judge evaluations as a PyTest CI gate so regressions are caught before deployment rather than in production
- ✓Self-hosting observability inside a regulated environment where traces containing customer data cannot leave the network
- ✓Automated prompt optimisation from production traces using the Agent Optimizer's six algorithms against a labelled dataset
- ✓Teams already using Comet for classical ML experiment tracking who want LLM observability in the same platform
Not Ideal For
- ✗TypeScript-first engineering teams, since the JavaScript and TypeScript SDK is materially less mature than the Python one
- ✗Teams with no infrastructure capacity, because the self-hosted stack runs on ClickHouse and needs at least 8GB RAM, with Kubernetes documentation thinner than the Docker Compose path
- ✗Buyers who weight community size heavily; Langfuse has a larger star count and a more generous free cloud tier at 50,000 spans against Opik's 25,000
- ✗Teams expecting Agent Optimizer to work out of the box, as it needs pre-labelled datasets that most projects do not have on day one
Integrations
Deployment
Market & Ratings
Comet reports 150,000+ developers and roughly 450 enterprise, startup and academic teams, with over 40 million traces processed daily; named users include Netflix, Uber, Etsy, Shopify, NatWest, Ancestry, AssemblyAI and Zappos
Market Analysis
Pros
- ✓Genuinely open source: Apache-2.0, self-hostable in full, with unlimited spans and seats and no feature gating against the paid tiers
- ✓Unusual breadth in one tool, covering tracing, LLM-as-a-judge evaluation across 30-plus metrics, prompt management, guardrails and CI integration
- ✓60-plus framework integrations plus OpenTelemetry, so it drops into an existing agent stack without rewrites
- ✓Cheap paid tier at $19 per month for 100,000 spans and up to 50 seats, well below per-seat enterprise observability pricing
- ✓Recognised in the February 2026 Gartner Market Guide for AI Evaluation and Observability Platforms
Cons
- ✗Smaller community than Langfuse, and the free cloud tier is tighter at 25,000 spans per month against Langfuse's 50,000
- ✗TypeScript and JavaScript SDK maturity trails Python, a real constraint for TS-first application teams
- ✗Self-hosting brings a ClickHouse backend requiring at least 8GB RAM, and Kubernetes documentation is noticeably thinner than the Docker Compose path
- ✗Agent Optimizer only pays off with pre-labelled datasets, which most teams do not have when they first adopt it
- ✗Independent practitioner discussion is thin: the original Show HN drew 86 points and 15 comments in September 2024 and every follow-up post has gone almost unnoticed, so production war stories are scarce
- ✗MLflow's own comparison guide rates Opik's agent tracing only 'moderate' and argues its strongest fit is teams already invested in Comet's ecosystem
Pricing
Open Source (self-hosted)
$0
- ✓Apache-2.0 licence
- ✓Unlimited spans
- ✓Unlimited team members
- ✓Full observability and agent testing feature set
- ✓Docker Compose or Kubernetes/Helm
Free Cloud
$0
- ✓25,000 spans per month
- ✓Up to 10 team members
- ✓60-day data retention
Pro Cloud
From $19/mo
- ✓100,000 spans per month
- ✓Up to 50 team members
- ✓Customisable retention
- ✓Additional spans at $5 per 100k
Enterprise
Contact for pricing
- ✓Unlimited spans and members
- ✓Flexible deployments
- ✓SSO, service accounts and view-only users
- ✓Dedicated support and SLAs
- ✓SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR
Metered in spans, not seats, which is the key commercial difference from per-user observability tools. The self-hosted Apache-2.0 build is genuinely unrestricted: unlimited spans, unlimited team members and the full feature set, with no gating between it and the paid cloud tiers, so the real cost is the infrastructure to run ClickHouse. Comet's cloud starts free at 25,000 spans per month for up to 10 members with 60-day retention; Pro is $19 per month for 100,000 spans and up to 50 members, with overage at $5 per additional 100,000 spans. Only operational and compliance capabilities are gated behind Enterprise, namely SSO, service accounts, view-only users, dedicated SLAs and the SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR attestations. Note the free cloud tier is half Langfuse's 50,000 spans.
Security & Compliance
Connect
Sources
This page was written from 7 sources, 3 on domains other than comet.com.
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
Latitude
Open-source observability and evaluation for AI agents — find and fix failures before production
Devin
The autonomous AI software engineer that plans, writes, tests, and ships production code inside your codebase
Gitar
AI code review that fixes your code instead of just flagging it
Maxim AI
Simulate, evaluate, and observe AI agents to ship 5x faster