These three products do not meter the same thing, and that single fact moves your bill further than any feature on any comparison page. Datadog charges for LLM spans only. LangSmith charges per trace. Arize AX charges per span and per ingested gigabyte. Run one agent request — three model calls, nine tool and retrieval steps — through all three and you have bought 3 billable units, 1 billable unit, or 12 units plus storage. Choose the metering axis that matches how your agents actually fan out. Then argue about dashboards.
The verdict: if you already run Datadog, turn on its Agent Observability for the operational layer and put a second, OpenTelemetry-native tool beside it for quality. Datadog's meter is the friendliest one for agents and it is the only product here that puts an LLM call in the same trace as the Postgres query that fed it. It is not the product that tells you the answer was wrong. For that second layer, default to self-hosted Langfuse. Take Arize AX instead if you also run classical ML models or need HIPAA-ready self-hosting. Take LangSmith only if your team lives inside LangGraph and can accept pricing you cannot model in advance.
| Datadog Agent Observability | Arize AX | LangSmith | Langfuse (self-hosted) | |
|---|---|---|---|---|
| Billing unit | LLM spans only | Every span + ingested GB | Whole traces | Every unit (trace, observation, score) |
| Published entry price | $160/mo for 100K LLM spans, annual | $50/mo Pro: 50K spans + 10 GB | $39/seat/mo + 10K base traces | $0 licence (MIT) |
| Overage rate published? | Yes — $3.50 per extra 10K LLM spans | No | No | Yes — $6.00–$8.00 per 100K units |
| Retention on entry price | 15 days | 30 days | 14 days (base traces) | Your storage, your call |
| Instrumentation standard | Datadog SDK on dd-trace | OpenInference on OpenTelemetry | LangSmith SDK / OTel | OpenTelemetry |
| Classical ML drift | No | Yes | No | No |
| PII redaction happens | In Datadog's backend | Your VPC if self-hosted | Client-side, before send | Wherever you run it |
| Self-host | No | Enterprise, Kubernetes | Enterprise add-on, phones home unless air-gapped licence | Yes, no gate |
The Billing Unit Decides Your Bill, Not the Feature List
An agent trace is mostly not LLM calls, so a platform that bills every span charges you four times what a platform that bills only LLM spans charges for identical telemetry. This is the whole comparison and almost nobody leads with it.
Define the workload once, and hold every vendor to it. One million agent runs a month. Each run makes three LLM calls and nine non-LLM steps — a retrieval, a handful of tool calls, an embedding — for twelve spans and roughly 2 KB of payload per span. That is one million traces, three million LLM spans, twelve million total spans, about 24 GB ingested. It is a mid-sized production agent, not a hyperscaler.
Datadog is the only vendor of the three that will price that off a public page. Its published price list meters "LLM spans": $160 a month billed annually for the first 100K, then $3.50 per additional 10K. Three million LLM spans is 290 increments over the base, so roughly $1,175 a month at 15-day retention — and the nine non-LLM spans in every run cost nothing, because Datadog's own span taxonomy defines seven span kinds (LLM, workflow, agent, tool, task, embedding, retrieval) and the price list only names one of them. Retention is the trap: 90-day trace retention is a separate line at $4 per 10K LLM spans per month, which on this workload adds about $1,200, roughly doubling the bill. The product page confirms the free tier at 40K LLM spans and 15-day retention.
Langfuse bills the opposite way and still lands close, because its rates step down with scale. A unit is any ingested event — a trace, an observation, or a score — so twelve million observations plus a million traces is roughly thirteen million units. On the published graduated rates ($8.00 per 100K from 100K–1M, $7.00 from 1M–10M, $6.50 from 10M–50M) on the $199 Pro plan, that is about $1,100 a month with three years of data access. Self-hosted, the licence is $0 and you are buying ClickHouse capacity and an on-call rota instead.
Arize AX and LangSmith both stop publishing exactly where this buyer needs a number. Arize's pricing page lists Free at 25K spans and 1 GB, Pro at $50 for 50K spans and 10 GB with 30-day retention, and Enterprise as "Custom" — with no overage rate for spans or GB anywhere on the page. LangSmith's pricing page gives Developer at 5K base traces and Plus at $39 per seat with 10K, then the words "pay as you go" and a calculator; the billing docs confirm the allowances and the 14-day/400-day retention split but send you back to the pricing page for rates, and the only unit rates actually printed — $1.50 per LangChain Compute Unit, $1.00 per LangChain Storage Unit — are normalised compute and storage units, not a per-trace price.
That is not a scandal. It is a procurement fact: two of these four cannot be modelled before you talk to a salesperson, and on a page whose entire job is to help you decide, that asymmetry is the finding.
A "Trace" Means Three Different Things Here
A trace is the record of one request through your application, but each vendor draws its boundaries differently, and the boundary is what you pay for.
LangSmith is the cleanest to reason about. Its observability concepts define a run as "a single unit of work executed by an agent, such as calling an LLM, formatting a prompt, or retrieving documents," and a trace as "a collection of runs for a single operation." One user request equals one trace no matter how many runs it contains — which makes LangSmith structurally the cheapest meter for deep, recursive agents and the most expensive for a chatbot that makes one call per turn.
Datadog inverts it. A trace there is the same idea — "the work involved in processing a request in your LLM application" — but the billable atom is the LLM span, so the agent that loops fifty times through the same tool is nearly free while the one that calls three models is not. Arize sits on OpenInference, an Apache-2.0 set of conventions the project describes as "complimentary to OpenTelemetry to enable tracing of AI applications," and bills the raw span count — the most faithful representation of what happened, and the meter most exposed to agents that fan out.
The standards question underneath this matters more than the vendors want it to. The OpenTelemetry project's own GenAI observability post describes the gen_ai.* conventions as "already in use today and under active development." Active development is not stable: attribute names can still move between versions. So OTel-native instrumentation buys you real portability today and a migration you will have to do again later. That is still a better trade than a proprietary SDK, and it is the reason to instrument through OTel even if you land on Datadog.
Where Evals Stop Being a Feature and Start Being the Product
An eval is a scored, repeatable test of whether an output was correct — and it is the only thing on this page that answers a question your existing APM cannot even ask. Latency, error rate and token count tell you the system ran. None of them tell you it was wrong.
Datadog ships the thinnest bench of the three. Its managed evaluations documentation covers Language Mismatch and Sensitive Data Scanning, with custom LLM-as-a-judge evaluators available for everything else. Cost tracking is genuinely good — Datadog "automatically calculates an estimated cost for each LLM request, using providers' public pricing models and token counts annotated on LLM/embedding spans," per its cost monitoring docs — but cost is an operational metric, not a quality one.
Arize and LangSmith both treat the dataset-experiment loop as the centre of the product rather than an add-on, and that is the actual reason to run a second tool. Arize's open-source Phoenix — 10.9k GitHub stars, Elastic License 2.0, "built on top of OpenTelemetry and is vendor, language, and framework agnostic" — gives you tracing, evals, versioned datasets, experiments and a prompt playground with no feature gates and no usage cap, which makes it the cheapest way in this entire comparison to find out whether you need the paid tier at all. LangSmith's dataset and annotation tooling is the best developer loop of the four if your application is already LangGraph, and pretending otherwise would be dishonest.
The trap to avoid: an eval suite pinned to a model version is a wasting asset. When a provider quietly reships a checkpoint, your eval will not notice unless you have the dataset and the re-run automated. Whichever tool you pick, the artifact you are actually buying is the versioned dataset — not the dashboard.
Only Arize Detects Drift, and Only for Models You Already Had
Drift detection means comparing today's data distribution against a reference baseline and alerting when they diverge — and among these three, only Arize AX genuinely does it. Its drift tracing docs describe baselining against either training data, "to identify how your model changes between your features, predictions, and actuals," or historical production data for short-term drift, with embedding drift for unstructured inputs on top.
That machinery is inherited from Arize's classical ML observability business, and it is the single strongest reason to pay for AX over anything else here. It is also the reason most LLM-only buyers should not pay for it. If your production estate is an LLM application and nothing else, feature drift monitors have nothing to monitor. If you run fraud models, pricing models, or a recommender alongside your agents, Arize is the only product in this comparison that puts both on one pane, and that consolidation is worth real money.
Datadog and LangSmith substitute LLM-as-a-judge scoring over a rolling window for statistical drift. That catches quality regression, which is usually what you actually care about. It does not catch a shifted input distribution before the quality regression shows up. If Gartner's observability mandate lands the way it is written, that distinction stops being academic and starts being an audit finding.
Your Prompts Leave the Building Before Datadog Redacts Them
This is the one section a regulated buyer should read twice, and it inverts the ranking above.
Datadog's Sensitive Data Scanner can scan Agent Observability traces including LLM inputs and outputs, with three redaction actions available for them — redact, partially redact, hash. But the documentation is explicit about where the work happens: content is "scanned and redacted in the Datadog backend during processing." The raw prompt, with whatever the customer typed into it, crosses your perimeter and reaches Datadog before anything is removed. Redacting on-premises is possible, and it is a separate product — the Sensitive Data Scanner processor inside Observability Pipelines, which redacts "before they leave your premises."
LangSmith does the opposite by default. Its masking documentation puts the work in the SDK: LANGSMITH_HIDE_INPUTS=true and LANGSMITH_HIDE_OUTPUTS=true as environment variables, hide_inputs/hide_outputs callables on the client for selective redaction, and a regex anonymiser for emails, names, phone numbers, cards and SSNs — all executing on your machine before anything is transmitted. For a healthcare or banking workload that is not a nicer default. It is a different legal posture.
Arize and Langfuse resolve it by relocation. Arize's self-hosted AX states that "all observability data lives entirely within your own environment — your Kubernetes cluster and your own persistent/object storage under your security controls," deployable on GCP, Azure, AWS, Oracle Cloud, OpenShift, K3s, Rancher and Tanzu, with SOC 2 Type II, ISO 27001 and HIPAA-ready claimed. Langfuse self-hosted has the same property with no licence conversation at all.
One detail to pin down before an air-gapped deployment: self-hosted LangSmith phones home by default. The Kubernetes install guide requires egress access to https://beacon.langchain.com for licence verification, alongside Postgres, Redis and ClickHouse, at least 16 vCPUs and 64 GB of memory, and a licence key available only as an Enterprise add-on. That egress is escapable, but not by you: the egress documentation states billing telemetry "cannot be disabled" and that if you "need to run without any egress," you must "contact your account team about an offline (air-gapped) license" — and an offline deployment "cannot run Engine, because it cannot reach LangSmith Intelligence." So "no outbound connections from this cluster" is achievable here, at the cost of a second negotiation and a feature you give up permanently.
This Does Not Replace Your APM. It Also Does Not Replace Your Evals.
For a platform team that already owns APM, the honest answer is that LLM observability adds exactly two things: semantic spans and scored outputs. Everything else — latency percentiles, error budgets, service maps — you already have and are already paying for.
Datadog is the only one of the three where those two layers are the same system rather than two systems you correlate by hand. Its APM correlation docs note that "the Agent Observability SDK is built on APM's dd-tracer," so a Python service already running dd-trace links its LLM spans to its APM spans "without additional setup." Use the HTTP API instead and you are capturing span IDs and threading them through requests yourself. That single integration is worth more in a 2 a.m. incident than any eval feature in this article, because the question at 2 a.m. is "which layer broke," and it is answered by one trace or by three browser tabs.
But being the APM is also why Datadog's quality story is the weakest. The same argument played out when Cisco bought Galileo and folded it into Splunk: the platform vendors are buying the semantic layer because they cannot grow it, and a bought layer takes years to feel native. Meanwhile the reason most agent pilots die on the way to production is not that nobody watched the latency graph.
Run both. Instrument once through OpenTelemetry so the second one is replaceable.
Who Should Not Buy Each of These
Do not buy Datadog Agent Observability if you are not already a Datadog customer. Standing this up as your first Datadog product means adopting an SDK on dd-trace and a pricing surface with retention as a separate multiplier, to get an eval bench thinner than the free tier of two competitors. Also skip it if your compliance position requires redaction before egress and you are not also buying Observability Pipelines.
Do not buy Arize AX if your production estate is LLM-only. You are paying for drift machinery built for tabular and embedding models you do not have, on the least agent-friendly meter in this comparison — every span, plus gigabytes — with no published overage rate to model against. Run Phoenix free instead and revisit when you have classical models in production, or a HIPAA requirement that forces self-hosting.
Do not buy LangSmith if you are not committed to the LangChain stack. It is the deepest framework gravity of the four, its self-hosting is Enterprise-gated and phones home for licence checks unless you separately negotiate an offline licence and surrender Engine, and — for a buyer whose job is to forecast a budget — you cannot compute next quarter's bill from the public page. Its trace-level meter is genuinely the cheapest unit for deep agents; that advantage is real and it does not survive being unable to price it.
Do not self-host Langfuse if you do not already operate ClickHouse. The licence is free; the ClickHouse cluster, the Postgres instance, the backups and the person who gets paged are not. On this workload the Cloud plan is around $1,100 a month, which is less than a fractional SRE.
Four Questions That Predict Which One You Regret
- What fraction of your spans are LLM calls? Under 30% and Datadog's meter is a structural discount nobody else offers. Over 70% and the metering axis stops mattering, so decide on evals instead.
- Do prompts contain regulated data? If yes, backend redaction is a compliance conversation, not a settings page. That eliminates SaaS Datadog on its own or forces Observability Pipelines into the deal.
- Do you have non-LLM models in production? If yes, Arize consolidates two budgets into one. If no, you are buying an ML platform to watch a chatbot.
- Can you leave in a quarter? Instrumented through OpenTelemetry, yes. Instrumented through a vendor SDK, you are re-instrumenting every service. This is the question that costs the most and gets asked the least.
What changes the answer: Datadog publishing a real eval bench, LangChain publishing a per-trace rate, or Arize publishing an overage rate. Two of those are documentation changes. Watch for them.
What to Do in the Next 30 Days
This week: Count your spans. Instrument one representative agent, run a hundred real requests, and record the ratio of LLM spans to total spans and the average payload size. Every quote in this article is a function of those two numbers, and almost no team knows them before the renewal conversation.
This month: Run Phoenix or Langfuse locally against that same agent — both are free and neither needs a procurement cycle — and build the versioned eval dataset. Then price all four vendors against your measured span ratio, not their example workloads, and ask Arize and LangChain in writing for the overage rate and the per-trace rate respectively. A vendor that will not put a unit rate in an email is telling you something about the renewal.
Before your next renewal: Get the redaction boundary in writing. Ask each vendor exactly where PII is removed — in your process, at the collector, or in their backend — and have your security lead sign the answer. That sentence, not the feature matrix, is what a regulator will ask about.
The Bottom Line
The LLM observability market is repeating the APM market's first decade: a platform incumbent with the best correlation story and the worst domain depth, specialists with real depth and no distribution, and an open-source layer quietly becoming the default substrate underneath all of them. ClickHouse acquired Langfuse in January 2026 and committed to keeping it "100% open-source under its existing MIT license for core features which allows for self-hosting at production scale" — a project already at 20,000+ GitHub stars and, by ClickHouse's own claim, trusted by 63 of the Fortune 500. LangSmith self-hosted runs on ClickHouse too. So does Datadog's competition. The storage layer has already been decided.
Which means the durable decision here is not the vendor. It is the instrumentation. Emit OpenTelemetry, keep your eval datasets in your own repository, and the observability vendor becomes a line item you can renegotiate instead of a migration you have to fund. Every team that skipped that step in the APM era paid for it twice.
Pick the meter that matches your agents. Own the instrumentation that outlives the pick.
Continue Reading
- Pinecone vs Weaviate vs pgvector: Stay on Postgres
- LangGraph vs CrewAI vs AutoGen: One of Them Is Retired
- Snowflake Cortex vs Databricks Mosaic AI: Pick on Exit Cost
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- Datadog Just Bought the Key to Self-Improving Enterprise AI
- AI Observability Engineering: Why Traditional Monitoring Misses 90% of Agent Risks
- Claude vs GPT vs Gemini: Stop Comparing Per-Token Prices