Every tool in this comparison can show you a runaway agent. Exactly one of them can stop it. Six of the seven sit beside the request path and read traces after the fact; the stop button lives in your runtime, your gateway, or a separate security product you have not bought yet. So buy the tracer for the audit trail and the replay, then wire the kill path yourself — and wire it first. For the workload below, self-hosted Langfuse is the right answer for most teams: it is the only MIT-licensed option here, it costs nothing to run on your own hardware at any volume, and its cloud rate is the cheapest of the deep-tracing options at agent-shaped workloads.
What We Priced: One Agent, 50,000 Runs, 12 Steps
Every vendor on this list quotes a different unit — traces, spans, gigabytes, requests, scores — so the only honest comparison normalises them to one workload. Ours is a single production agent doing 50,000 runs a month at an average of 12 steps per run: roughly 600,000 model calls, about 2.4 million spans and scores, and around 4.8 GB of trace payload at 2 KB per span. Ten engineers need access. All prices were read from the vendors' own pricing pages on 11 August 2026.
| Tool | Billed on | Cost at this workload | Can it stop a run? | Self-host | Replay a bad run |
|---|---|---|---|---|---|
| Langfuse | traces + observations + scores | ~$199/mo cloud, $0 self-hosted | No | Yes, MIT | Add any observation to a dataset, re-run |
| LangSmith | traces + $39/seat | $390/mo in seats; per-trace rate not published | Yes — but via the LangGraph runtime, not the dashboard | Enterprise only | Yes: checkpoint time travel |
| Braintrust | GB ingested + scores | $249/mo — $1,074 if you score every step | No | Enterprise only | Yes: log → dataset → experiment |
| Datadog Agent Observability | LLM spans | ~$160 entry tier (third-party figure); overage rate not published | No — blocking is a separate product (AI Guard) | No | No |
| Arize Phoenix | nothing (self-hosted) | $0, Elastic License 2.0 | No | Yes, ELv2 | Yes: spans → datasets → experiments |
| W&B Weave | GB ingested, $0.10/MB | ~$390/mo | No | Enterprise (custom) | Yes |
| Helicone | requests, through a proxy | $79/mo + usage | Yes — it is in the request path | Yes, Apache-2.0 | No |
Two columns in that table decide the purchase, and neither is price. One is "can it stop a run". The other is "self-host", because the audit trail of an agent that touches production systems is exactly the data you least want to rent.
A Read-Path Tracer Cannot Stop Your Agent
A monitoring platform is a read path. It receives telemetry after a step has already executed, which means it is structurally incapable of preventing the next one. This sounds obvious written down and is comprehensively obscured by the category's marketing, which uses "guardrails", "protection" and "runtime" for features that score outputs after the model has produced them.
Check it against the documentation. Datadog's own LLM Observability docs describe tracing, anomaly detection, sensitive-data scanning and prompt-injection identification — no blocking or halt primitive appears anywhere in the observability product. Its cost documentation is about estimating what your model provider charges you, using "providers' public pricing models and token counts annotated on LLM/embedding spans". That is a bill you read, not a bill you cap.
Datadog does sell the missing primitive — as a different product, on a different meter. AI Guard is a security layer that evaluates prompts, model responses and tool calls inline, and once you trust your policies you "can enable blocking of steps that are flagged as unsafe", including a tool blocklist. Note what that concedes: it sits "in the critical path", it is a Security SKU rather than an observability one, and it is still Limited Availability with the coding-agent variant in preview. The lesson is not that Datadog is better than the table suggests — it is that even the vendor with the deepest observability stack had to ship the stop button somewhere else entirely.
The controls that actually stop an agent live one layer down, in the harness:
- Step limits. LangGraph raises
GraphRecursionErrorwhen a graph exceeds its recursion limit, and the documentation's own advice — pass "a higherrecursion_limitvalue into yourconfigobject", with{"recursion_limit": 1000}as the worked example — is the thing you must not do reflexively in production. - Turn limits. The OpenAI Agents SDK raises
MaxTurnsExceededwhen a run passesmax_turns, alongside tool timeouts and input/output guardrail tripwires. - Cancellation. LangGraph Platform exposes a cancel endpoint with two semantics worth knowing before you need them:
interrupt"stop[s] the worker executing the run and marks the run asinterrupted" while keeping checkpoints for audit, androllback"removes it and its checkpoints from storage" and reverts thread state. One of those destroys your evidence. - Spend caps in the request path. Only something sitting in the request path — a proxy, a gateway, an inline guard — can refuse a call. That is the whole architectural argument for putting an LLM gateway in front of your agents.
Raising the ceiling is not a fix, and the framework's own issue tracker shows why. A LangGraph 1.0.6 user reported a text-to-SQL agent looping until it hit the recursion limit, re-attempting variations of the same failed query rather than halting, and noted the same configuration stopped correctly on 0.6.x. The issue was closed as not planned. A step limit is a backstop against your framework's behaviour, not just your prompt's.
The cost of skipping this is not theoretical. A widely recirculated account of a four-agent research system describes two agents locked in a recursive clarification loop for eleven days and $47,000, with no crashes, no timeouts and no alerts — the system looked healthy from the outside the entire time. Treat that one as a parable rather than a case study: no company is named in any retelling. The documented end of the scale is Uber, which capped employees at $1,500 a month per agentic coding tool after burning its annual AI budget in four months. Neither problem is a dashboard problem.
Langfuse Wins Because You Can Take the Meter Off Entirely
Langfuse is the default pick because its worst case is free and its cloud case is the cheapest deep-tracing option at agent-shaped volumes. The repository is MIT licensed "except for the ee folders", and the self-hosting guide gates only a short list behind a licence key — organization creators, the instance-management API, UI customisation. Nothing in that list is tracing, evals or retention. You can run the whole product on your own infrastructure and keep every prompt, tool call and argument inside your boundary, which is the posture an agent audit trail deserves.
On cloud, a billable unit is "any tracing data point sent to the platform — including traces, observations… and scores", per the pricing page. Our 2.4 million units on the $29 Core plan work out to roughly $170 in graduated overage ($8 per 100k up to 1M, $7 per 100k above it) — about $199 a month, with unlimited users. No seat tax is the quiet advantage here: it means your risk and compliance people can read the traces without a procurement conversation.
Be clear about what that meter does, though, because it is the same trap as everyone else's: an observation is a step. Double your agent's step count and you roughly double your Langfuse Cloud bill. What Langfuse gets right is the shape of the curve rather than immunity from it — graduated rates that fall as you grow, no per-seat multiplier, and no per-gigabyte cliff — plus the exit that none of the commercial options offer, because the self-hosted deployment meters nothing at all. If your step count is genuinely unpredictable, that exit is the feature.
For replay, Langfuse's answer is deliberately unglamorous and works: you + Add to dataset on any observation of a production trace, attach the expected output, and re-run experiments against it. The documented workflow is exactly the one you want — "select production traces where the application did not perform as expected".
Do not pick Langfuse if you need a mature alerting and anomaly engine out of the box, or if nobody on your team will own a ClickHouse-backed deployment. Self-hosting is free in licence and not free in attention.
Where LangSmith and Braintrust Actually Beat It
LangSmith is the only tool here whose vendor also ships the runtime, and that is the entire reason to buy it. If you are on LangGraph, you get time travel: get_state_history returns the checkpoint chain, you replay from a prior checkpoint, and update_state forks a branch — noting that it "does not roll back a thread. It creates a new checkpoint that branches from the specified point. The original execution history remains intact." That is a real debugging superpower, and it is a property of the framework, not the dashboard.
The pricing is where it gets awkward. LangChain's pricing page publishes $39 per seat on Plus, 10,000 included base traces, 14-day base retention against 400-day extended retention, and $1.50/LCU and $1.00/LSU for deployments — but no per-trace price. The billing documentation defines LCUs and LSUs as deployment compute and storage units and refers you back to the pricing page for trace rates. At ten seats you are at $390 a month before a single trace over the allowance, against a rate you cannot model in a spreadsheet. Get it in writing.
Braintrust is the pick if your problem is quality rather than uptime. Its logging guide is built on one structural idea — "Logs use the same data structure as experiments", so "production data seamlessly becomes evaluation datasets" — which closes the loop between an incident and the regression test that prevents it recurring. The trap is in the meter. Pro is $249 with 5 GB and 50,000 scores included, then $1.50 per 1,000 scores. Score once per run and you are at $249. Score every one of your 12 steps and you are at 600,000 scores, $825 in overage, and $1,074 a month — a 4.3x swing driven entirely by an evaluation design decision nobody makes in procurement.
Datadog Agent Observability: Right Answer If You Already Pay Datadog
Buy Datadog Agent Observability when the agent's failures are indistinguishable from your infrastructure's failures — and only then. Its genuine advantage over every specialist here is that the agent trace lands in the same pane as the database latency and the pod restarts, and it is backed by an alerting engine that has been hardened for a decade. When your agent's p99 blows out because a downstream service degraded, that correlation is worth more than any eval feature.
Two things to price in. First, the pricing page lists the product but does not publish an on-demand per-span rate above the included tier, so — exactly like LangSmith — the number that determines your bill at 600,000 model calls a month is a sales conversation, not a published figure. Third-party teardowns quote entry pricing around $160 a month for 100,000 LLM spans; treat any per-span number you find on a blog as unverified until your account team confirms it. Second, that same page flags Agent Observability as "currently unavailable on the US-FED site" — which quietly removes it from consideration for a whole class of public-sector buyers.
Do not pick Datadog if you are not already a Datadog customer. The correlation advantage is the only reason it wins, and it evaporates the moment your infrastructure telemetry lives somewhere else.
W&B Weave Is the Loser at This Workload
W&B Weave loses this comparison on one number: $0.10 per megabyte. The Weights & Biases pricing page includes 1.5 GB of Weave ingestion on the $60 Pro plan and charges $0.10/MB beyond it — roughly $100 per gigabyte. Braintrust charges $3 per gigabyte for overage on the same kind of data. That is a 33x difference in the identical unit, and it lands hardest on precisely the workload this article is about, because agent traces are large: every tool call carries its arguments and its full result back into the payload.
At 4.8 GB a month our reference agent costs about $390 on Weave against $249 on Braintrust and $199 on Langfuse Cloud — and the gap widens linearly with every step you add to the agent. Weave is a reasonable choice for a team already standardised on Weights & Biases for model training, where the experiment lineage genuinely matters. It is the wrong meter for high-volume agent telemetry, and no amount of sampling makes a per-megabyte charge the right shape for a system whose defining characteristic is verbose intermediate state.
Arize Phoenix and Helicone are the two useful edge cases. Phoenix is free, self-hostable and OpenTelemetry-native, with tracing, evals, datasets and experiments — but it ships under the Elastic License 2.0, not an OSI-approved licence, which is a materially different conversation with your legal team than Langfuse's MIT. Moving up to hosted Arize AX means 25,000 spans a month free at 15-day retention, or $50 for 50,000 spans on Pro — about 2% of our reference workload, so anything real is an enterprise quote. Helicone is the odd one out and the most interesting: it is a proxy, so at $79 a month on Pro with unlimited seats it is the only tool in this comparison sitting in the request path, where rate limits can actually refuse a call. Its one-month retention on Pro makes it a control point, not an audit archive.
"OpenTelemetry-Native" Is Not Portability Yet
Every vendor here claims OpenTelemetry compatibility, and none of it yet guarantees you can leave. The GenAI semantic conventions were split into their own repository, open-telemetry/semantic-conventions-genai, and the specification for agent spans still carries "Status: Development" at the top of the document — not Stable. The operation vocabulary it defines is genuinely the right model for agents (create_agent, invoke_agent, execute_tool, plan, retrieval, plus a full set of memory operations), but a convention in Development can change attribute names between releases.
What that means practically: your instrumentation is portable, your dashboards, saved views and alert rules are not, and a vendor's OTel support tells you nothing about whether they will export your historical traces on the way out. Ask for the export format and the retention terms in the contract, the same way you would for any AI platform where the exit cost is the real cost.
This also matters for compliance. If your agent is in scope as a high-risk system, EU AI Act Article 12 requires that it "technically allow for the automatic recording of events (logs) over the lifetime of the system" — the lifetime, which is a longer horizon than Helicone's one month, Datadog's default window, or Langfuse Cloud's 90 days on Core. Retention tier is a compliance decision that gets made by accident during a pricing negotiation.
Who Should Not Buy Each of These
The honest version of a buyer's guide is the exclusions:
- Not Langfuse if you want alerting and anomaly detection to work on day one, or nobody will own the deployment.
- Not LangSmith unless you are on LangGraph. Without the runtime you are paying $39 a seat and an unpublished trace rate for tracing that Langfuse does for free.
- Not Braintrust if you are going to score every step of a 12-step agent — model that bill before you sign, not after.
- Not Datadog Agent Observability if your infrastructure telemetry is somewhere else, or if you are a US federal buyer.
- Not Arize Phoenix if your legal team treats Elastic License 2.0 as a commercial licence, which many do.
- Not W&B Weave for high-volume agent traces at $0.10/MB, unless you are already deep in Weights & Biases.
- Not Helicone as your system of record — one month of retention on Pro is a control plane, not an audit trail.
Wire the Kill Path Before You Buy the Dashboard
This Week: Find out what your agents' current step ceiling is. If you are on LangGraph and nobody has set recursion_limit, it is 25 and it is the only thing between you and an eleven-day loop. Set an explicit turn cap on every agent and make exceeding it page someone.
This Month: Put a hard daily spend ceiling in the request path — a gateway or a proxy, not an alert. An alert tells you about the $47,000 after it is spent; a proxy refuses call number 4,001. Confirm that whoever holds the on-call pager can cancel a running agent from a terminal in under 60 seconds, and that your cancel semantics preserve checkpoints rather than deleting them.
Before Your Next Renewal: Price your actual span volume against your actual meter. Take one week of production traces, count traces, spans, scores and gigabytes separately, and run all four numbers through each vendor's published rates. Where a rate is not published — Datadog's per-span overage, LangSmith's per-trace price — get it in the contract before you commit, and cap the annual increase. Then check that your retention tier matches your regulatory obligation and not your budget.
The Bottom Line
The agent monitoring market is selling forensics as if it were control, and the distinction is the whole ballgame. This is the same shape as the gap between agent permissions and agent policy: every step is logged, every call is authorised, and the system still does the wrong thing for eleven days because nothing in the loop had the authority to say no.
Self-host Langfuse. Keep your traces inside your boundary, pay nothing for the licence, and accept the operational tax. Add Braintrust later if quality regression is your real problem, or Datadog if your agent's failures are really infrastructure failures wearing a hat. But do the cheap thing first: set the step limit, put a spend cap in the request path, and make sure someone can kill a run at 2am.
A dashboard that shows you the fire is not a fire suppression system. Buy the extinguisher first.
Continue Reading
- Datadog vs Arize vs LangSmith: Buy on the Billing Unit
- Best LLM Gateways for Cost Control: Self-Host First
- LangGraph vs CrewAI vs AutoGen: One of Them Is Retired
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- Claude Code Stops Asking Aug 14. Prompts Aren't Policy.
- Meta's Agent Model Fits on a Laptop. Nothing Logs It.
- Best RAG Platforms for Regulated Industries: Permissions First
- One Agent Escalated Another. Every Call Was Authorized.