If your release gate trusts an LLM judge you picked on price or because it came bundled with your eval tool, you are probably trusting a number nobody has measured. On JudgeBench, an ICLR 2025 benchmark of hard response pairs labelled by objective correctness, GPT-4o on a plain prompt scored about as well as a coin flip. A reasoning model scored 81%. Every fine-tuned judge except Skywork landed below the random baseline.
The verdict: for any gate that checks correctness, use a frontier reasoning model as a pairwise judge, from a different model family than the system you are grading, with the answer order swapped and a reference answer in the prompt. Then measure it against 100 to 200 items a person has labelled before you let it block a release. Use a panel of cheap models only for subjective checks, and only after it passes the same calibration. Do not put a fine-tuned open judge in front of a correctness gate.
| Judge option | What you would buy today | What JudgeBench measured for the class | Cost per judged item (workload below, Oct 25, 2026) | Verdict |
|---|---|---|---|---|
| Frontier reasoning model | GPT-6 Sol, Claude Opus 5.5, Gemini 3.1 Pro | 73% to 81% (o3-mini high, o1-preview, DeepSeek-R1) | $0.032 Sol, $0.064 Opus, $0.036 Gemini Pro | Default for correctness gates |
| General model, no reasoning | Any model with thinking turned off | 51% to 64% (GPT-4o, Claude 3.5 Sonnet) | n/a | Format and tone checks only |
| Cheap small model | GPT-6 Luna, Claude Haiku 5.5, Gemini 3.8 Flash | Not tested on JudgeBench | $0.0016 Luna or Haiku, $0.012 Flash | Unknown until you calibrate it |
| Cross-family panel, majority vote | Luna + Haiku 5.5 + gpt-oss-120B | Not tested; a published panel beat a single GPT-4 judge | $0.0053 | Good for high-volume subjective checks |
| Fine-tuned open judge | Skywork-Critic 70B, Prometheus 2 | 13% to 57%; all but Skywork under 50% | Your GPU bill | The loser for correctness |
| Reward model | Skywork-Reward-Gemma-2-27B | 59% to 64% | Your GPU bill | Best-of-N ranking, not a gate |
| Multi-agent debate | ChatEval on GPT-4o | 34% | Several calls per item | Avoid |
The workload for every cost figure: a 2,000-item release gate, pairwise, each pair judged twice with the order swapped (4,000 calls), 3,000 input tokens and 1,000 output tokens per call, thinking tokens billed as output. Prices are from each vendor's live pricing page, checked October 25, 2026.
What JudgeBench Actually Measured
JudgeBench measures whether a judge can pick the objectively correct answer between two responses on hard knowledge, reasoning, math and coding questions. The project repository holds 350 pairs generated by GPT-4o and 270 generated by Claude 3.5 Sonnet. Each pair has one correct and one incorrect response, so the label comes from ground truth rather than from a person's preference.
The scoring rule is stricter than it looks. Per the paper, each judge sees every pair twice with the order swapped, and "inconsistent decisions (e.g., A>B in one trial, A<B in the other) or ties in both trials are deemed incorrect." The random baseline is 50%. A judge that changes its answer when you swap the order gets marked wrong, and that rule explains how a judge can score below a coin flip.
The published numbers, overall accuracy:
- Prompted judges: GPT-4o on a vanilla prompt, 50.86%. GPT-4o with the Arena-Hard judge prompt, 56.57%. Gemini 1.5 Pro through Vertex AI Evaluation, 44.57%.
- The same Arena-Hard prompt on stronger models: o3-mini at high reasoning effort, 80.86%. o3-mini at medium, 76.57%. o1-preview, 75.43%. DeepSeek-R1, 73.14%. Claude 3.5 Sonnet, 64.29%.
- Fine-tuned judges: Skywork-Critic Llama 3.1 70B, 57.43%. Skywork 8B, 53.43%. Prometheus 2 8x7B, 40.29%. AutoJ, 36.57%. JudgeLM 33B, 35.71%. PandaLM, 13.14%.
- Reward models: Skywork-Reward-Gemma-2-27B, 64.29%. InternLM2-20B-Reward, 63.43%.
- Multi-agent: ChatEval on GPT-4o, 34.00%.
The authors' summary: "All of our fine-tuned judges (except Skywork) perform significantly below the random baseline."
The detail that should change your purchase is o3-mini. It is a small, cheap model (listed at $1.10 per million input tokens and $4.40 output on OpenAI's pricing page, October 25, 2026), and it beat every non-reasoning judge in the table. Moving from medium to high effort added 4.3 points. In this table, how much the judge thought tracked accuracy on hard pairs more closely than its size or its price did.
One caveat cuts the other way. JudgeBench's table predates GPT-6, Claude 5.5 and Gemini 3.x. Nobody has published those models' JudgeBench scores in the paper, so the class result is the evidence and your own calibration set is the proof.
Which Judge Should You Use for a Release Gate?
Use a frontier reasoning model with its effort turned up for any gate where a wrong answer is the failure you are trying to catch. That covers factual accuracy, code correctness, policy compliance and math. The options, one at a time.
Frontier reasoning model
This is GPT-6 Sol, Claude Opus 5.5 or Gemini 3.1 Pro with thinking on. At the workload above, Sol costs $0.016 per call ($2 input, $10 output per million tokens on OpenAI's page), so $0.032 per judged item and $64 per 2,000-item release. Opus 5.5 is $4 and $20 per million (Anthropic pricing), so $0.064 per item and $128 per release. Gemini 3.1 Pro Preview is $2 and $12 under 200K tokens (Google pricing), so $0.036 per item and $72 per release. All three vendors take 50% off for batch jobs, and a release gate rarely needs an answer in under a minute.
Skip it if you are judging millions of production samples a day for monitoring. At that volume the bill is real money, and a sampled reasoning judge plus a cheap panel on the rest is the better split. Also skip Gemini 3.1 Pro for a gate you need to hold steady for a year: Google still labels it Preview.
Cheap small model
GPT-6 Luna (OpenAI) and Claude Haiku 5.5 both list at $0.10 input and $0.50 output per million (Haiku's price rises 5x for prompts over 100,000 tokens, per Anthropic). That is $0.0016 per judged item, or $3.20 per release. Gemini 3.8 Flash is $0.75 and $3.75 through December 31, 2026, doubling to $1.50 and $7.50 on January 1, 2027, per Google's page. Budget for the 2027 price.
These are the right price for format, tone, length and "did it refuse" checks. No published JudgeBench number exists for them, so their accuracy on your correctness questions is unknown until you measure it.
Do not use one to check correctness without a calibration set. A cheap judge that agrees with your labellers 80% of the time on easy items can still miss the hard ones, and the hard ones are the regressions you shipped the gate to catch.
Cross-family panel
A panel is three or more judges voting independently. The PoLL paper found a panel of smaller models from "disjoint model families" outperformed a single large judge, showed "less intra-model bias," and was "over seven times less expensive." A Luna, Haiku 5.5 and gpt-oss-120B panel costs about $0.0053 per item at our workload, using gpt-oss-120B at $0.15 and $0.60 per million on Together AI (pricing page, October 25, 2026).
Keep the panel independent. JudgeBench's ChatEval, where GPT-4o agents debate before deciding, scored 34%, worse than any single prompted judge in the table.
A panel is the wrong pick for a team without the plumbing to run three providers, parse three outputs and break ties. Three integrations also means three deprecation schedules to track.
Fine-tuned open judge
Prometheus 2 is Apache 2.0 and supports both 1-to-5 grading and pairwise ranking against a rubric and reference answer. Skywork-Critic 70B posts 93.3 on RewardBench, and Skywork's two critics were the only fine-tuned judges to clear 50% on JudgeBench. The 70B model ships under the Skywork Community License rather than a standard open licence, so put it through legal review.
This is the loser. Apart from Skywork, the class scored below chance on hard pairs. Eugene Yan's review of the evaluator literature found fine-tuned evaluators behaved like task-specific classifiers, and "applying them to a different scheme (e.g., direct scoring) led to a catastrophic performance drop." The same review reports them scoring worse than random on the LLMBar fairness benchmark. The upside is millisecond latency and data that never leaves your network, which matters for a privacy constraint and does nothing for accuracy.
Newer open judges trained with reinforcement learning to reason before they decide do better than this class. Meta's J1 paper reports J1-Llama-70B at 60.0% on JudgeBench, above every fine-tuned judge in the original table, but still below o1-mini's 64.2% in the same comparison.
The one team that should pick it anyway cannot send data to an outside API, fine-tunes the judge on its own labels, and calibrates the result on its own domain.
Reward model
A reward model returns a scalar score in one forward pass with no explanation. Skywork-Reward-Gemma-2-27B scored 64.29% on JudgeBench, above every fine-tuned generative judge. Ai2's RewardBench 2 moved to a best-of-4 format (random is 25%), and models score about 20 points lower on it, on average, than on the original. Use a reward model to rank candidates in best-of-N sampling. It gives a release reviewer nothing to read when it blocks a build.
How to Calibrate the Judge Against People
Calibration means measuring how often the judge agrees with a person on items from your own product before you trust any score it produces. Build a set of 100 to 200 items, have one domain expert label each pass or fail with a written reason, and run the judge on the same items.
Hamel Husain's guide gives the working numbers: about 100 labelled examples per failure mode, because below 60 "confidence intervals are often too wide." He recommends binary pass/fail over 1-to-5 scales, one principal domain expert rather than a committee, and separate dev and test splits so you do not tune the prompt on the items you score it with. In his Honeycomb example it took three prompt iterations to reach over 90% agreement with the expert.
Report at least two numbers. Raw agreement flatters a judge when most items pass. Husain reports true positive and true negative rates separately. Cohen's kappa corrects for chance: in one study in Yan's review, Llama 3 8B agreed with humans 80% of the time and had a kappa of only 0.62. The best evaluators in that review reached kappa of 0.84 (GPT-4) against 0.97 between humans.
The Judging the Judges study adds the failure mode to look for: judges with high percent agreement "can still assign vastly different scores," and they lean lenient. If your judge passes items your expert failed, it is too lenient for a gate, whatever its agreement rate.
The labelling is the expensive part. One study cited in Yan's review paid $8 per human-labelled sample, so a 150-item set costs about $1,200 in annotation. That is about nine releases of Opus 5.5 judging at our workload, and no other spend tells you whether the judge works.
Which Judge Biases Break a Release Gate, and the Fixes
Three biases are documented well enough to design around: position, verbosity and self-preference. The MT-Bench paper named all three while showing GPT-4 reached over 80% agreement with human preferences.
Position bias means the judge prefers whichever answer sits in a given slot. Yan's review reports claude-v1 showed it about 70% of the time on MT-Bench. Fix it the way JudgeBench does: run every pair in both orders and treat a flipped verdict as a tie or a fail, never as a win.
Verbosity bias is a preference for the longer answer. The same review reports claude-v1 and GPT-3.5 picked the longer, padded answer more than 90% of the time. Put a reference answer in the prompt and tell the judge to grade against it, which is the format Prometheus 2 was trained on.
Self-preference is a judge favouring its own outputs. Panickssery, Bowman and Feng found models can recognise their own text, and after fine-tuning they measured a linear correlation between that recognition and how strongly the model favoured itself. If your product runs on Claude, judge it with GPT-6 Sol or Gemini, or a panel with no Claude model in it.
Pairwise comparison generally beat direct scoring in a study Yan reviewed, though the gap narrowed on objective tasks. For a release gate, compare the candidate build to the last shipped build on the same input, with a reference answer, in both orders.
What Changes the Answer
Four facts about your gate predict regret better than the judge's leaderboard rank.
- Does the gate check correctness or style? Correctness needs a reasoning judge. Style can run on a calibrated cheap model or panel.
- Is the judge in the same family as the system it grades? If yes, switch families. Self-preference is measured, and it makes your own regressions look like improvements.
- Can the data leave your network? If not, a fine-tuned open judge or a self-hosted open-weight reasoning model is your only option, and calibration stops being optional.
- What is your judge's version policy? A judge upgrade can move every score with no change to the system you grade.
On the last point, pin the exact model ID. Anthropic states that every Claude model ID is a pinned snapshot, including the dateless IDs from the 4.6 generation on, and lists a retirement date for each. Pinning delays the problem until retirement, so re-run the calibration set on every judge version change, and record which judge version produced each baseline. Scores from two judge versions are not comparable until both have been checked against the same human labels.
Public judge benchmarks share one gap: a judge that scores well on public pairs can still fail on your domain. JudgeBench's questions are math, code, knowledge and reasoning. Your insurance claims summaries and contract clauses are not in it.
What to Do Before the Next Release
This Week:
- Write down which model your gate's judge uses, its exact version ID, and whether it shares a family with your production model.
- Pull 150 real items from production logs and have your domain expert label each pass or fail with a one-line reason.
This Month:
- Run your current judge on those items and compute true positive rate, true negative rate and Cohen's kappa. If kappa is below 0.6 or the judge passes items the expert failed, the gate is not doing its job.
- Re-run the same set on a cross-family reasoning judge with order swapping and a reference answer. Move to it if it wins, and use the batch API to halve the cost.
Before the Next Judge Upgrade:
- Add the calibration set to your eval tool (Braintrust, Langfuse or your own harness) as a judge test that runs whenever the judge's model ID changes, and block the upgrade if agreement drops.
The Bottom Line
LLM judges are where A/B testing was before teams learned to check for sample ratio mismatch: a number on a dashboard that everyone trusts and nobody audits. The evidence favours reasoning judges, cross-family grading and order swapping, and it rules out fine-tuned open judges for correctness. It does not tell you how any judge does on your product. Label 150 items and run them before your next release decision.
Continue Reading
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- RAG Eval Dataset: 200 Hand-Written Questions Beat Synthetic Ones
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.
- Only 3 of 36 Model Gaps Were Real. Size Your Eval Set.
- Vendor AI Benchmarks Need Independent Runs With Task Logs
- Model Deprecation: Bedrock Keeps Claude 4 Months Past Anthropic
