Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate

The regression gate is the free part. Promptfoo's CLI returns exit code 100 the moment one test case fails, under an MIT licence, with your prompts never leaving the machine. What the paid platforms sell is everything after the gate — the stored trace, the run-to-run diff, the review queue a non-engineer can use, and the judge that watches live traffic.

That distinction is the whole buying decision, and most comparisons blur it. If what you need this quarter is a thing that blocks a bad prompt from merging, you can have that today for nothing. If what you need is a place to keep 90 days of scored production traffic and let three people argue about a rubric, you are buying a database with a UI, and you should price it like one.

Here is the shortlist, checked against each vendor's live pricing and docs on 27 August 2026.

	Promptfoo	Langfuse	Opik	Braintrust
Licence	MIT	MIT core; EE key for RBAC, audit logs, masking	Apache-2.0, full backend	Proprietary
Fails the build by default	Yes — exit code 100	No — you raise RegressionError	No — you write the pytest assert	No — the action reports, you write a Reporter
Scores live production traffic	No	Yes	Yes	Yes
Dataset versioning	Your git history	Automatic, every edit makes a version	Datasets + experiments	Immutable experiment snapshots
Metered on	nothing	units (traces + observations + scores)	spans	scores + GB ingested
Cheapest paid tier	contact sales	$29/mo Core	$19/mo Pro	$249/mo Pro
Self-host on a self-serve plan	Yes	Yes	Yes	No — Enterprise only
600k scores/month (derived)	$0	~$115	~$44	~$1,074

The recommendation: gate on Promptfoo, keep on Langfuse or Opik, and buy Braintrust only when your bottleneck is human review throughput rather than tooling. Promptfoo is the only one of the four that fails a build without you writing the code that says no. Langfuse and Opik are both genuinely self-hostable on a plan you can buy with a credit card. Braintrust has the best cross-run review experience of the four and prices it on the one axis you want to grow.

---

What You Are Actually Buying Is Storage and Review

An eval framework does four separable jobs, and every product here charges for a different subset. Naming them is what stops you from paying twice.

The first is running a fixed dataset against a candidate prompt and producing a score — an eval, in the narrow sense. The second is turning that score into a merge decision. The third is keeping the runs so you can diff today against last Tuesday. The fourth is scoring live production traffic on a sample.

Job one and job two are commodity. A YAML file, a CLI, and a non-zero exit code do it, and Promptfoo does exactly that for free. Jobs three and four are where the money is, because they are storage and query problems, and storage and query problems have a marginal cost. That is why Braintrust meters "processed data" in GB and evaluation "scores" by the unit, why Langfuse meters "units", and why Opik meters spans.

Read the meter before the feature list. It tells you what the vendor thinks it is selling.

Only One CLI Fails the Build Without Custom Code

Promptfoo is the only one of the four where "block the merge" is the default behaviour rather than something you implement. This matters more than it sounds, because the code nobody writes is the code that does not run.

The Promptfoo docs are explicit: "The eval command will return exit code 100 when there is at least 1 test case failure or when the pass rate is below the threshold set by PROMPTFOO_PASS_RATE_THRESHOLD." The threshold defaults to 100%, and PROMPTFOO_FAILED_TEST_EXIT_CODE lets you change the code. You add one step to a workflow file and the build goes red.

Every other option ships a reporter and asks you to build the enforcer:

• Langfuse publishes langfuse/experiment-action, whose should_fail_on_regression input defaults to true — but it only fails when your script raises RegressionError. The documented pattern is that you read the average accuracy, compare it to a constant you chose, and raise. The action respects your decision; it does not make one.
• Braintrust publishes braintrustdata/eval-action, which posts a live summary comment with 🟢 and 🔴 markers next to changed scores. Its terminate_on_failure input stops the process on an error, not on a regression. Per Braintrust's own CI guide, bt eval "returns a non-zero exit code if any eval throws an exception," and enforcement comes from a custom Reporter() whose reportRun(results) returns true or false.
• Opik ships a pytest integration, which is a reasonable answer — pytest already fails builds — but the threshold is still an assert you wrote.

None of that is a defect. A team that wants a soft launch, where the eval posts a comment for six weeks before it blocks anything, is better served by a reporter. But be honest about which one you are installing. A gate that reports is a dashboard.

The practitioner guidance here is consistent and worth copying: a two-layer design with a curated 40-to-100-case gate on every pull request and a broader nightly matrix, blocking on prompt-template, model-name, tool-schema and guardrail changes, with zero critical failures allowed and a 98% pass rate on high-severity cases. "More cases do not automatically mean better coverage." A small suite everyone respects beats a large one everyone reruns until it passes.

The Same 600,000 Scores Cost $0 or $1,074

Normalise to one workload or the price sheets are incomparable. Take a 400-case regression suite with three scorers per case, run 500 times a month — every pull request plus a nightly. That is 600,000 recorded scores and roughly 200,000 traced case-runs.

Option	Meter	Derived monthly platform bill
Promptfoo Community	none	$0
Langfuse self-hosted	none	$0 + your infrastructure
Opik self-hosted	none	$0 + your infrastructure
Opik Cloud Pro	600k spans	~$44
Langfuse Cloud Core	1.2M units	~$115
Langfuse Cloud Pro	1.2M units	~$285
Braintrust Pro	600k scores	~$1,074

The arithmetic, from published rates:

• Braintrust Pro is $249/month and includes 50,000 scores, with overage at $1.50 per 1,000, per the pricing page. The other 550,000 scores cost $825. The 1.2 GB ingested sits inside the 5 GB allowance.
• Langfuse counts "any tracing data point sent to the platform — including traces (complete application interactions), observations (individual steps: spans, events, and generations), and scores (evaluations)". At one trace, two observations and three scores per case that is six units, so 1.2M units. Graduated overage is $8.00 per 100k up to 1M and $7.00 per 100k after, which is $86 on top of the $29 Core fee.
• Opik Cloud Pro is $19/month with 100k spans included and $5 per 100k after, so 500k of overage is $25.
• Promptfoo charges nothing for the eval itself. Enterprise adds team sharing, continuous monitoring, SSO and a compliance dashboard at an unpublished price.

Two things are worth saying plainly. First, judge-model tokens are roughly the same bill in all four cases and often exceed the platform fee — Braintrust Pro at least bundles $249 of model credits against that, which materially narrows the gap if you route the judge through its proxy. Second, Braintrust's own Starter plan is worse than Pro at this volume: no platform fee, but 10k included scores and $2.50 per 1,000 after works out near $1,476. The crossover is around 200,000 scores a month. Below that, Starter; above it, Pro.

The structural objection stands regardless. Braintrust meters the thing it wants you to do more of. Add a fourth scorer to the suite and the bill rises 28%. That is a bad incentive to install inside a quality process, and it is the same shape as the per-span trap in AI observability pricing.

Dataset Versioning: Git Is a Legitimate Answer

Every option here versions your eval set; they just disagree about where it lives. There is no wrong answer, only a wrong assumption about who owns the file.

Langfuse versions automatically: "Every add, update, delete, or archive of dataset items produces a new dataset version," retrievable by timestamp, so you can rerun last month's experiment against last month's dataset. That is the strongest story of the four, and it matters most when non-engineers edit cases in a UI — which is exactly when a git-based workflow breaks down.

Braintrust's model is the immutable experiment: a run is a snapshot, and the product is built around diffing two of them row by row. Opik pairs datasets with experiments the same way.

Promptfoo's answer is that your dataset is a YAML or CSV file in your repository, and your versioning is git log. Reviewers can see a test case change in the same diff as the prompt change that motivated it. For an engineering-owned suite this is better, not worse — and it is the reason the tool needs no dataset service at all. It stops working the day your compliance lead wants to add twelve cases without opening a pull request.

Nobody Calibrates Your Judge, Including These Four

All four ship LLM-as-a-judge scorers. None of them tells you whether yours agrees with your humans, and that is the single largest source of false confidence in this category.

An LLM-as-a-judge scorer is a prompt that receives the input, the output and a rubric, and returns a score with reasoning — Langfuse describes it exactly that way. What none of these products measures for you is the agreement between that score and a human's on the same rows.

The failure modes are documented and they do not shrink as models improve. A 2026 study across 20 mainstream LLMs found self-preference bias — judges favouring their own outputs — and reported that "advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB." Upgrading the judge model does not fix the judge. Position bias and verbosity bias sit alongside it.

The work is yours regardless of vendor, and it is a week, not a quarter:

1. Sample 100-300 real traces, stratified across the outcomes you care about.
2. Have two or three people label them against the written rubric, and compute inter-annotator agreement first. If your humans do not agree with each other, the rubric is the problem, not the model.
3. Score the same rows with the judge and compute judge-to-human agreement on the same scale.
4. Re-run that agreement check every time you change the judge model — a lesson the corpus keeps re-teaching, most recently when a post-training refresh silently invalidated an agent eval and when prompt techniques stopped paying after a model version upgrade.
5. Use a judge from a different model family than the system under test, and run both orderings on pairwise comparisons.

Grade the suite pass/fail against explicit criteria rather than tracking an average, which is the same error that made agent benchmark averages look healthy while only 30% of runs were usable.

Self-Hosting Is the Line Braintrust Paywalls

Three of these four can run entirely inside your network on a plan you can buy without a sales call. Braintrust cannot, and for a regulated buyer that is the decision.

• Promptfoo is the strictest: "LLM evals run 100% locally - your prompts never leave your machine," per the project README. Nothing to deploy, nothing to egress.
• Langfuse is MIT licensed except for its ee folders, and per its open-source docs "All product capabilities—tracing, evaluations, prompt management, experiments, annotation, the playground, and more—are MIT licensed without any usage limits." The enterprise licence key gates project-level RBAC, protected prompt labels, data retention policies, audit logs, server-side data masking, SCIM and the org management API — which is a fair split, though note that three of those are the things a security review asks about.
• Opik is Apache-2.0 including the backend, deployable by Docker Compose or Helm, with no commercial licence required for the full platform. On licence terms alone it is the most permissive of the four.
• Braintrust offers a genuine hybrid split — the data plane holds "all sensitive data, including experiment records, logs, traces, spans, datasets, and prompt completions" in your cloud, while Braintrust runs the UI and auth, and "Braintrust's servers and employees do not require access to your data plane for it to operate." Two caveats: org-level LLM provider secrets sit encrypted in Braintrust's control plane, and health and system telemetry flows back by default. The larger caveat is commercial — hybrid and self-hosted are Enterprise-plan features, listed on the Enterprise tier of the pricing page and on neither Starter nor Pro. On Starter and Pro, the two tiers you can actually self-serve, your prompts and outputs go to Braintrust's cloud.

Count the operational cost honestly on the other side. Self-hosting Langfuse means running Postgres, ClickHouse, Redis or Valkey, S3-compatible object storage, a web container and a worker container. That is a platform team's Tuesday, not a developer's afternoon. If nobody owns that stack, $29 a month for Langfuse Cloud Core is the cheaper answer and you should stop arguing about licences.

Where the Eval Tool Stops and Observability Starts

The boundary is production traffic, and it is the cleanest line in this category. Promptfoo is on one side of it; the other three cross it.

Promptfoo has no production trace store and does not sample live traffic. Its scope ends at the pull request, which is why it costs nothing and why it cannot be your only tool. Langfuse and Opik are observability platforms with evals layered on: Langfuse runs judges on incoming traces with rule-based filtering and sampling, and Braintrust logs production alongside experiments and charges the same per-score rate either way.

That crossing is where the bill changes character. A 400-case gate is a fixed cost you control. Production spans are a variable your traffic controls, and agentic workloads multiply spans per user action. If you route production traces into the same tool as your gate, you have merged a predictable line item with an unpredictable one, and the per-span, per-retention pricing traps apply in full. Opik makes this vivid: ingest is $5 per 100k spans, but retention beyond 60 days is $29 per 100k spans — storing costs almost six times ingesting.

None of these four is your runtime control either. An eval framework tells you a change is worse; it does not stop a running agent. That needs monitoring with an actual kill switch, and it is a separate purchase from a separate budget. If you are still choosing the layer underneath, score the orchestration platform on exit terms first.

Who Should Not Buy Each One

The useful half of a comparison. Each of these is a real disqualifier, not a caveat.

Do not pick Promptfoo if you need production traces scored on a sample, a review queue that a compliance or support lead will actually open, or a published enterprise price you can put in a procurement pack — all paid tiers are contact-sales. Its collaboration story at the free tier is a shared results URL. It is a Node CLI, and a team that lives in notebooks will fight it.

Do not pick Langfuse Cloud Core if you need enterprise SSO or RBAC — that is the Pro tier plus a $300/month Teams add-on, or Enterprise at $2,499/month, which buys audit logs, SCIM and an uptime SLA rather than capacity, since the included 100k units is identical across Core, Pro and Enterprise. Do not self-host Langfuse if no named team will own ClickHouse in production.

Do not pick Opik Cloud if long retention matters, at $29 per 100k spans beyond 60 days. Its cloud free tier caps at 10 team members and 25k spans a month.

Do not pick Braintrust if data residency or self-hosting is a hard requirement and you are not buying Enterprise; if your score volume is high and your budget is fixed; or if you specifically want the CI integration to enforce rather than report. Do pick it when your constraint is human review throughput — the experiment diff and review workflow is the thing the free tools genuinely do not have, and the bundled model credits offset a real part of the fee.

Four Questions That Predict Regret

Ask these in the trial, not after the invoice. They are the ones that separate teams who kept the tool from teams who quietly stopped running it.

1. Does the gate fail closed by default, or does somebody have to write the code that says no? If it is the second, name that person before you sign.
2. What is the meter, and does it charge more when you evaluate more thoroughly? Model the bill at three times your current suite size, because that is where you are going.
3. Can you self-host on the plan you can actually buy? "Available on Enterprise" means "available after a sales cycle and a minimum commit."
4. Who owns judge calibration, and what triggers a re-check? The correct trigger is any judge-model change, not a calendar quarter.

What changes the answer: if a non-engineer owns the eval set, the git-file model breaks and you need a dataset service. If you are under an audit obligation that requires retained scored records, the storage bill is unavoidable and you should optimise it rather than resent it. And if your suite is under about 50,000 scores a month, all four are close enough to free that you should choose on workflow.

What to Do in the Next 30 Days

This week: Stand up Promptfoo against 50 curated cases — 20 happy path, 15 edge, 10 refusal, 5 tool-call — and wire promptfoo eval into the pull-request workflow with a required status check. Cost: an afternoon and zero dollars. Do this even if you intend to buy something, because it establishes the baseline the paid trial has to beat.

This month: Run the judge calibration. Sample 200 production traces, get two humans to label them, and compute agreement twice — human-to-human first, then judge-to-human. Write the number down. It is the only figure in this whole exercise that tells you whether any of the scores mean anything.

Before your next renewal cycle: Model the platform bill at three times your current score volume against each vendor's published rate, and separately price the production-trace side. If those two numbers land in the same contract, split them deliberately rather than by accident. Ask every shortlisted vendor for the self-host terms in writing on the tier you intend to buy, not the tier the salesperson would prefer.

The Bottom Line

The last decade of software testing settled this argument once already. The thing that runs the tests and returns a non-zero exit code became free and boring, and the money moved to the systems that store, correlate and let people argue about the results. LLM evaluation is arriving at the same place faster than the price sheets have adjusted.

Take the free gate now. It is genuinely good, it is MIT, and it works this afternoon. Then buy storage and review as storage and review, with the volume modelled and the meter understood — and refuse to pay a premium that scales with the number of questions you ask about your own system.

The eval is not the product. The decision to stop the merge is the product, and nobody sells you the discipline to enforce it.

Continue Reading

• AI Observability Pricing: Same 10 GB, $49 or $930
• Best AI Agent Monitoring: Langfuse, Then a Real Kill Switch
• Datadog vs Arize vs LangSmith: Buy on the Billing Unit
• Few-Shot Stopped Paying on GPT-4o. Qwen Still Wants It.
• RAG Build vs Buy: Buy the Index. Build the Eval Set.
• Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
• DeepSeek Swapped the Model. Your Eval Didn't Notice.
• Agent Orchestration Platforms: Score Exit, Not Features