If you need a hallucination score on customer-facing RAG answers this quarter, start with a prompted LLM judge on a cheap, pinned model, and make Cleanlab TLM earn its place on your own labeled data. LLM-as-a-judge beat open Vectara HHEM on all six datasets in the main published comparison, at $0.74 per 1,000 checks using the study's model. Use HHEM-2.1-Open only where answers cannot leave your network.
No tool in this comparison can reliably tell you that the document it checked against is out of date. Budget for retrieval evals either way.
| LLM-as-a-judge | Cleanlab TLM | Vectara HHEM | |
|---|---|---|---|
| Verdict | Start here | Bake off, then decide | Loser for correctness; keep for no-egress |
| What it is | A rubric prompt to a model you already pay for | A scoring wrapper around a base LLM, using multiple calls | A trained classifier: premise in, score out |
| AUROC, six datasets (vendor-linked study) | 0.618 to 0.837 | 0.755 to 0.943 | 0.450 to 0.725 |
| Cost, 1M checks a month (November 3, 2026) | $500 to $10,000 by model | Rates shown only inside your account | Open weights plus your compute; platform from $100K a year |
| Self-hosting | Only if you self-host the judge model | Private deployment on enterprise plans | Open weights run locally on CPU |
| Do not pick it if | You have no labeled answers to set a cutoff | You need a public price or a written roadmap today | Your answers involve arithmetic, or you would score summaries at a 0.5 cutoff |
How Do Cleanlab TLM, Vectara HHEM and an LLM Judge Differ?
A hallucination detector is a model or method that scores a generated answer, in real time, without a reference answer to compare it to. The three options get to that score differently, and the difference decides what each one can see.
LLM-as-a-judge is the simplest. You send the context, question and answer to a model along with a scoring rubric, and it returns a rating. The study this page leans on used a five-point scale. The obvious weakness, which the paper states, is that you are asking the same class of unreliable system to grade itself.
Cleanlab's Trustworthy Language Model wraps a base LLM. The study describes it as combining self-reflection, consistency across sampled responses and probabilistic measures, and it takes your original RAG prompt with no special template. Cleanlab's FAQ says scores come from "multiple (non-deterministic) LLM calls" and does not give the number.
Vectara's Hughes Hallucination Evaluation Model is a classifier. You give it a premise and a hypothesis, and it returns 0 to 1, where 0 means the premise gives no evidence for the hypothesis. HHEM-2.1-Open is Apache-2.0, finetuned from google/flan-t5-base, and tagged English only. The newer HHEM 2.3 is commercial and available only through Vectara's platform.
What the Evidence Shows, and Who Ran It
The published evidence favors TLM, but the study behind it has close ties to Cleanlab. The paper, arXiv 2503.21157, lists a single author at San Jose State University. The same study runs on Cleanlab's blog credited to that author and Jonas Mueller, a Cleanlab co-founder. Its code sits in Cleanlab's GitHub organization, and its FinQA dataset, credited to the same author, sits in Cleanlab's Hugging Face organization. Read it as vendor-linked work with reproducible code, which is useful and short of independent.
AUROC (area under the ROC curve) measures how often a detector ranks a wrong answer below a right one. 0.5 is a coin flip and 1.0 is perfect. All methods ran at default settings, and both TLM and the judge used gpt-4o-mini:
| Dataset | TLM | LLM judge | Prometheus 2 | HHEM 2.1 | Lynx 70B |
|---|---|---|---|---|---|
| FinQA | 0.868 | 0.789 | 0.604 | 0.450 | 0.702 |
| ELI5 | 0.755 | 0.618 | 0.731 | 0.540 | 0.575 |
| FinanceBench | 0.850 | 0.765 | 0.608 | 0.528 | excluded |
| PubmedQA | 0.885 | 0.786 | 0.854 | 0.602 | excluded |
| CovidQA | 0.943 | 0.837 | 0.852 | 0.725 | excluded |
| DROP | 0.886 | 0.739 | 0.512 | 0.490 | excluded |
(Lynx, from Patronus AI, was excluded where it had seen the data in training. The paper also dropped the rest of the HaluBench suite, citing annotation errors. Figures come from the paper's ROC charts, published in April 2025.)
The judge beat HHEM on every row, and HHEM fell below chance on FinQA and DROP, which need reasoning and arithmetic. Cleanlab has a commercial interest here, since the paper's conclusion favors wrappers like TLM over custom-trained models, so discount the size of that gap. The direction mostly holds in the independent test below, with an exception on QA. The strongest defense of HHEM is that it was built to check factual consistency, while this study labeled overall correctness, so it was graded on a wider job than it was trained for. That defense is fair, and it is also why HHEM alone is the wrong check for a customer answer.
TLM's lead over the judge, 0.079 to 0.147 AUROC, is the claim to test yourself. It came from the vendor-linked study, on one base model, 19 months ago.
The closest independent test complicates the ranking. A June 2026 Lund University master's thesis, carried out at Robert Bosch AB with no detector vendor among its authors or supervisors, ran HHEM-2.1-Open and prompted judges on 404 RAGTruth samples labeled for faithfulness to the context, the job HHEM was built for. A G-Eval judge on gpt-4o-mini, using a prompt the authors refined by hand, scored 0.770 AUROC against HHEM's 0.724. On GPT-5.2 the same judge scored 0.852, while Ragas on GPT-5.2 scored 0.708, below HHEM, so both method and model move the result. On QA, the task closest to a support answer, HHEM's 0.760 beat the gpt-4o-mini judge's 0.700, on about 135 samples per task type, which the authors call indicative. If your traffic is mostly QA, run HHEM beside the judge from the start. Open-weight LettuceDetect scored 0.809 overall, though it was fine-tuned on RAGTruth and the authors warn that may inflate its score, so add it to any local-only test.
A separate UCLA preprint, with no vendor on the paper, measured HHEM on HaluEval: 67.2% of hallucinations caught and 86.6% of good QA answers cleared, for 76.9% accuracy. On summaries it caught 32.2%, which the authors attribute to scoring the whole summary as one unit. The paper cites HHEM 2.1 without naming the checkpoint it ran or the cutoff it chose. Lund put HHEM's summary AUROC at 0.758 and its MCC-tuned cutoff at 0.727, well above the 0.5 Vectara uses, so much of the summary weakness may be a cutoff problem. We found no independent head-to-head of all three options, because neither test included TLM.
Vendor-run results need the same label. Vectara reports that HHEM-2.1 beats GPT-3.5 and GPT-4 judges on balanced accuracy and F1, using 2024 models, though it says Ragas faithfulness on GPT-4 can match it on F1. Its September 2025 comparison is more useful because it cuts against its own free product: on RAGTruth summaries, HHEM-2.1-Open's median score on hallucinated answers was at least 0.643, above the post's 0.5 cutoff, so it called most of them factual. HHEM-2.3's recall there was at least 0.758. None of the studies above reports testing 2.3.
What Does a Checked Answer Add in Latency and Cost?
A judge on your answer model nearly doubles the bill, and a small judge adds about 5% to 7%. The workload below is ours: 1M answers a month, 4,000 input tokens and 300 output tokens per answer, then a 4,500-token judge prompt with a 100-token reply. Rates come from OpenAI's pricing page, checked November 3, 2026, without caching or batch discounts.
| Line item | Rate per 1M tokens (in / out) | Per 1,000 | Per month |
|---|---|---|---|
The answer itself, gpt-6-sol |
$2.00 / $10.00 | $11.00 | $11,000 |
Judge on gpt-6-sol |
$2.00 / $10.00 | $10.00 | $10,000 |
Judge on gpt-4o-mini (the study's model) |
$0.15 / $0.60 | $0.74 | $735 |
Judge on gpt-6-luna |
$0.10 / $0.50 | $0.50 | $500 |
The study's scores do not transfer to gpt-6-luna, so re-test before you switch to save $235 a month. Include one larger model in that test, since the Lund thesis saw its G-Eval judge gain 0.082 AUROC moving from gpt-4o-mini to GPT-5.2.
TLM's price is hard to budget. The FAQ offers free tokens and then pay-per-token, with rates visible only under Usage & Billing in your account. Because each score uses multiple LLM calls, assume it costs more than one judge call on the same base model until your invoice says otherwise. Its tuning guide says the default configuration "is not latency/cost-optimized" and that the default preset is medium. We found no published seconds-per-score figure, so measure it.
HHEM-2.1-Open has no per-call fee. Vectara's model card puts it at about 1.5 seconds for a 2k-token input on a modern x86 CPU, using under 600MB of RAM. If time scales linearly, which is our assumption, a 4,300-token check takes about 3.2 seconds, or 3.2M CPU-worker seconds a month. A month has about 2.6M seconds, so plan on more than one always-busy worker on average, plus headroom for peaks. The commercial route starts high: Vectara's pricing page, checked November 3, 2026, lists SaaS from $100K a year, VPC from $250K and on-prem from $500K. That is a platform purchase, and it makes no sense for the scorer alone.
Vectara's own figures put Ragas with GPT-4 at up to 35 seconds for a 4,096-token context, which is vendor-run and uses an older model. The Lund thesis measured medians of 18.54 seconds for Ragas on gpt-4o-mini, 2.52 seconds for its G-Eval judge on that model and 0.34 seconds for HHEM, on samples far shorter than our 4,300 tokens. Whichever you pick, the pattern Cleanlab suggests for low latency works for all three: stream the answer first and fetch the score afterwards. Block delivery only on intents where a wrong answer costs money, such as refunds or contract terms.
What None of These Tools Catch: A Faithful Answer From a Stale Document
All three score an answer against the context you hand them, so a policy document retired last March can produce a high score on a wrong answer. Ragas faithfulness makes the limit explicit: it is the share of claims the retrieved context supports. Vectara's documentation likewise says its score measures accuracy against the search results.
A judge or TLM could in principle notice a claim that clashes with what the base model knows. Do not count on that. The ClashEval study found LLMs adopted incorrect retrieved content over their own correct prior knowledge more than 60% of the time, and no base model knows your internal refund policy. An RMIT and Deakin paper on medical RAG found lower ROUGE-1 scores for all five models when they got the most contradictory documents in place of the most similar ones.
So rely on retrieval evals first if your knowledge base churns weekly, if you have never measured whether the right document comes back, or if your support escalations trace to outdated articles. A scorer will not move those numbers. Build a hand-written eval set, pool your relevance judgments when you compare retrievers, and put an owner and expiry date on every source document.
What Does a Score of 0.7 Mean?
A score of 0.7 means something different on each tool, so never carry a cutoff from one to another.
On TLM, Cleanlab says: "We typically consider trustworthiness scores below 0.7 low," with scores above 0.9 high and the band between depending on your use case. It also warns that correct answers can score low when the model would likely get similar questions wrong. The tuning guide adds that magnitudes are not comparable across presets, so choose the preset before you choose a threshold.
On Vectara's platform, the docs call the score calibrated, so 0.7 reads as roughly a 70% likelihood of consistency, and suggest starting at 0.5. Check that calibration against your own labels, because it is Vectara's claim.
On a five-point judge, 0.7 is not a score at all. The model returns one of five ratings, so you have only four possible cutoffs, which is why the judge's curves in the study have so few bends. If you need finer control, you will have to change the prompt or add a second judge.
False positives are what get a detector switched off, and the math is unforgiving. Take the UCLA QA result, where 86.6% of good answers cleared means 13.4% were flagged, at a threshold already tuned for what the paper calls F1 (it defines that as the mean of the two rates). Assume 10,000 answers a day and that 5% are wrong (our illustration):
| Outcome | Calculation | Answers a day |
|---|---|---|
| Wrong answers caught | 500 × 67.2% | 336 |
| Wrong answers missed | 500 × 32.8% | 164 |
| Correct answers flagged | 9,500 × 13.4% | 1,273 |
| Share of flags that are real | 336 ÷ 1,609 | 20.9% |
About four of every five flags would be false, and 1,273 correct answers would land in a review queue daily. We found no published study on when support teams abandon these tools, so set your own floor. Ours: if fewer than half of flags are real on your labeled sample, route flagged answers to a softer action, such as a citation prompt or a "check this" notice, and keep humans out until precision improves. AUROC cannot tell you this. It ranks answers, and your queue depends on one operating point at your real error rate.
Where Your Answers Go
Data egress splits these options more cleanly than accuracy does. A judge adds no new vendor if it runs with the provider that already generates your answers, under the agreement you already hold. HHEM-2.1-Open sends nothing out, though its loading path uses trust_remote_code=True, so have security review the repository and pin a revision.
TLM sends prompts and responses to Cleanlab unless you buy private deployment, which it describes as running in your own cloud. The same FAQ says scores are cached and the cache refreshes every 30 days. The privacy policy available today is dated February 18, 2022 and does not mention API prompts or give a retention period for submitted content, so ask for those terms in writing.
Vendor continuity is the other question. Cleanlab announced its acquisition by Handshake on January 28, 2026. TechCrunch described it as primarily an acqui-hire, with nine key employees joining Handshake's research organization, and did not say what happens to product customers. The CEO's letter commits the open-source package to Apache-2.0 and says nothing about TLM's future. TLM's docs are still live, but that is all the assurance we could find.
Who Should Not Pick Each Option
Skip the prompted judge if you cannot pin a model version, since Vectara warns that a model update can flip an assessment, and that claim comes from a vendor with a reason to make it. Skip it too if you have no labeled answers, because an uncalibrated cutoff on a five-point scale is a guess. Skip it if using the answer model as judge would double a bill you cannot defend.
Skip TLM if procurement needs a public price, if you cannot get a written roadmap from Handshake, or if your data cannot reach a third party and private deployment is out of budget. A small team that wants a free, permanent building block has less reason to depend on it after the acquisition.
Skip HHEM-2.1-Open for anything involving arithmetic or multi-step reasoning (FinQA 0.450, DROP 0.490), for summaries scored whole at a 0.5 cutoff, and for non-English answers. If you must run locally, Ragas's FaithfulnesswithHHEM splits an answer into claims and scores each with HHEM. That still needs an LLM for the splitting step, which you would also have to host. Skip Vectara's platform entirely unless you want its retrieval and agent stack too, in which case the score comes with each query.
The loser is HHEM-2.1-Open used alone as the check on customer answers. It is free and fast, and it missed two-thirds of hallucinated summaries in the UCLA preprint and most of them at a 0.5 cutoff in Vectara's own post. Before you rule it out on QA traffic, test it, since it beat a gpt-4o-mini judge there in the Lund thesis.
What Predicts Regret, and What Changes the Answer
A threshold copied from a benchmark is the likeliest source of regret, because your own error rate sets your flag precision. Three findings would change our recommendation.
If your bake-off shows TLM beating the judge by a margin close to the study's (about 0.08 or more AUROC) and it clears your flag-precision floor, buy it, provided the roadmap and data terms come back in writing. Hard data-residency rules move you to HHEM with claim splitting or a self-hosted judge. And if most of your wrong answers trace back to the wrong document, stop shopping for a scorer and fix retrieval and reranking first.
This Week:
- Pull 200 production answers, stratified by intent, and have support leads label each as correct, wrong from the wrong document, or wrong despite the right document. The split between the two kinds of wrong answer tells you whether you have a hallucination problem or a retrieval problem.
- Run a pinned judge over them using the rubric in the paper, and log scores in the eval tool you already have, such as Langfuse or Braintrust.
This Month:
- Run TLM at
lowandmedium, plus HHEM-2.1-Open, on the same 200 answers using the public benchmark code. Record AUROC, flag precision at your real error rate, and p95 seconds per score. - Ask Cleanlab for token rates, API data retention terms and a written TLM roadmap under Handshake.
- Ship the winner in shadow mode for two weeks, with scores logged and nothing blocked, and compare flags against support tickets.
Before You Go Live:
- Set the threshold from the shadow data, with one owner for changing it. Re-run the label set whenever the judge or base model changes.
- Add an owner and expiry date to every source document, because no score here will catch an expired one.
The Bottom Line
A prompted judge at under $1 per 1,000 checks gets you a working baseline and, more usefully, the labeled set you need to judge any vendor. TLM may earn its premium, but only your labels can show that, since the evidence for it traces back to Cleanlab.
Label the 200 answers before you take a vendor call.
Continue Reading
- RAG Eval Dataset: 200 Hand-Written Questions Beat Synthetic Ones
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- Cohere vs Voyage vs BGE Rerankers: Voyage Wins on Most RAG Shapes
- What RAG Actually Costs: $1,308 a Month at 10M Tokens/Day
- RAG Build vs Buy: Buy the Index. Build the Eval Set.