The reason you have not re-tested your retriever is a number nobody on your team ever calculated. Four engineers at JPMorganChase calculated it and published the receipt: 62 retrieval configurations compared on a live financial-news question-answering system for about $800 of model spend, because they judged the union of what those systems retrieved once and reused 79.6% of the labels on everything that came after.
The paper — Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection, submitted 2 September 2026 by Max Nelson, Hanoz Bhathena, Aviral Joshi and Saket Sharma — is worth an hour of your architect's time for a second reason. The five best configurations the bank measured spanned 0.007 MAP. OpenAI, Cohere and an Apache-2.0 model from Nomic all landed inside that band. The leaderboard did not choose the winner. Latency and embedding cost did.
What $800 Actually Bought
$800 bought 766,350 graded relevance judgments covering 62 retrieval configurations, accumulated over four weeks. The corpus was roughly 300,000 news article pages indexed in OpenSearch. The benchmark was 303 queries, deliberately assembled from four different populations rather than one: 135 generated from semantic clusters, 91 from metadata-clustered articles, 50 templatic entity lookups, and 27 downsampled from real production traffic, per the paper's production case study.
The 62 configurations were not 62 vendors. They were five embedding families, three retrieval modes (dense, sparse, hybrid), two or three output dimensionalities each, three query formulations, and variants with and without query expansion. Every system retrieved the top 100 documents per query. The judge was GPT-4.1 (gpt-4.1-2025-04-14), grading each query-document pair on a three-point scale.
The arithmetic is the part to take to your CFO. At OpenAI's published GPT-4.1 rate of $2 per million input tokens and $8 per million output, 766,350 judgments came to roughly a tenth of a cent each. Compare that to what the same job costs with people. Ian Soboroff, who leads the Retrieval Group at NIST and runs TREC, describes the human version plainly: "a typical TREC track usually involves a team of six contractors working for 2-4 weeks" — plus training, monitoring, and purpose-built annotation software.
That is the whole reason your 2024 embedding decision is still in production. Not conviction. A cost estimate anchored on the human process, never revised.
A Judgment Is an Asset, Not a Line Item
The core move is definitional: a pooled judgment is a relevance label attached to a (query, document) pair rather than to a system, which means any system that later retrieves that document scores against a label you already own. Evaluate N systems independently and you need on the order of N × Q × K judgments — systems × queries × depth — with heavy redundancy, because competing retrievers surface overlapping documents. Pool them and you judge the union once.
Incremental expansion is what turns that from a one-time saving into an operating model. You build the pool from your current candidates. When configuration N+1 arrives, you retrieve with it and judge only the documents it contributes that are not already in the pool. Everything else scores for free.
The production numbers: 766,350 pooled judgments against 3,765,643 for the same comparison run independently — a 79.6% reuse rate and a 4.9x cost reduction. At the same per-judgment rate, the independent version would have run near $3,900. Against the four public benchmarks (FiQA, TREC-COVID, Natural Questions and FinRAGBench-V) with 11 dense, sparse and hybrid systems, the effect was steadier and slightly smaller: 650,421 pooled judgments versus 1,907,400 independent, with reuse between 65% and 67% per dataset.
There is a second, subtler benefit. Classical TREC pooling has a "hole" problem — a genuinely novel retriever surfaces documents nobody judged, and gets penalised for them. The bank's answer is that cheap judging removes the constraint that created the hole: because LLM judging is inexpensive, they judge every document any system retrieved, so nothing relevant goes unassessed. That matters directly if you are evaluating anything unusual against an inherited gold set.
If you are still deciding whether to own the retrieval layer at all, this changes the arithmetic in the build-versus-buy decision for enterprise RAG — the eval set was always the part worth building, and it just got an order of magnitude cheaper to maintain.
Three Vendors, Seven Thousandths of a Point
The most useful result in the paper is the one that says the vendor choice barely mattered. Here are the bank's top five configurations by MAP@100 — mean average precision, the metric that rewards putting relevant documents high in the ranking:
| Configuration | Embedding family | MAP@100 |
|---|---|---|
| hybrid-emb3-large-768 | OpenAI text-embedding-3-large | 0.461 |
| hybrid-emb3-large-256 | OpenAI text-embedding-3-large | 0.459 |
| hybrid-nomic-1.5-768 | Nomic embed-text-v1.5 | 0.454 |
| hybrid-nomic-v2-moe-768 | Nomic embed-text-v2-moe | 0.454 |
| hybrid-cohere-v4-1536 | Cohere Embed v4 | 0.454 |
Seven thousandths of a point separates first from fifth. Two of those five are open-weight: nomic-embed-text-v2-moe is Apache 2.0, 475M total parameters with 305M active, and it tied Cohere Embed v4 on a bank's own financial-news corpus.
And the bank did not deploy the winner. It deployed number two — the 256-dimension variant — writing that the choice was made "based on these results combined with latency and embedding cost considerations." Read that sentence again. The evaluation's job was not to pick a model. Its job was to show the top of the field was close enough that operational cost could decide.
The paper is candid about where that reasoning stops. Its own limitation section states that when systems are separated by less than 0.001 nDCG, neither human qrels nor LLM pseudolabels can reliably distinguish them. Bootstrap resampling of the human gold labels showed that 74% of pairwise ranking disagreements fell within the qrels' own sampling uncertainty, leaving only seven meaningful nDCG@10 ranking differences across 220 system pairs, and 97% of pairwise orderings preserved once that uncertainty is accounted for. Note the scope: that uncertainty analysis runs on the four public benchmarks, where human qrels exist to resample. No equivalent error bars were computed for the 62 production configurations, so the 0.007 spread at the top is a gap between point estimates, not a measured tie.
That gives you a defensible floor for a procurement conversation. When a vendor's deck claims a benchmark win of under 0.001 nDCG, the correct response is not scepticism about their honesty. It is that the measurement cannot resolve the difference at any price, including theirs. The same discipline applies to reading agent benchmarks where the headline score hides the distribution and to scoring coding agents on scope rather than pass rate.
The Findings That Had Nothing to Do With Embeddings
The four most operationally useful results in the study are about how queries are phrased and how retrieval is fused — not about whose vectors you buy. These are exactly the findings a public leaderboard cannot give you, because they depend on your corpus and your users.
- Instruction-framed queries hurt everything. Wrapping the user's question in instructional scaffolding cost a mean 0.039 MAP across all systems, by diluting the retrieval signal. If your application prepends a template to the query before it hits the index, that is a measurable tax you are paying today.
- Keyword reduction hurts dense retrieval more than hybrid — mean 0.013 MAP — which is a concrete argument for keeping BM25 in the fusion rather than going purely vector.
- Query expansion made things worse on well-formed queries: 0.424 MAP with it, 0.459 without. It functioned as a robustness mechanism against verbose input, not as a general improvement. If you turned it on because a blog post said to, turn it off and measure.
- Dimensionality matters far less under hybrid. Going from 256 to 768/1536 dimensions bought a mean 0.029 MAP for dense retrieval but only 0.012 under hybrid, because BM25 fusion compensates for reduced embedding capacity.
That last one is a storage bill. Dropping from 1536 to 256 dimensions cuts your vector index roughly sixfold, and under hybrid retrieval the bank measured the cost at just over a hundredth of a point. Anyone sizing capacity across Pinecone, Weaviate and pgvector — or watching what a production RAG pipeline actually costs per month — should treat dimensionality as a variable to test, not a spec to inherit from a model card.
Where This Breaks, and Who Says So
Pooled LLM judging is defensible for choosing among your own candidates and indefensible as a published score, and the strongest objection comes from inside the information-retrieval community. Soboroff's SIGIR keynote is titled, without hedging, "Don't Use LLMs to Make Relevance Judgments".
The sharpest empirical version of that objection is Charles Clarke and Laura Dietz's "LLM-based Relevance Assessment Still Can't Replace Human Relevance Assessment", and it names the failure mode you actually have to guard against: circularity. When every system in their experiment used the same LLM assessor as a re-ranking component, Kendall's tau against human judgments collapsed from 0.84 to 0.63 across the top 60 systems — and to −0.40 among the top five systems, with twelve systems scoring above 0.95 nDCG under automatic assessment against human scores of 0.68 to 0.72. They then submitted a run deliberately engineered to exploit the judge. It placed 5th under automatic assessment and 28th under human judgment.
Steel-manned properly, that is not an argument against the bank's method. It is a constraint on it: your judge model must not be the model you are also using to rank, re-rank, or generate. If you judge with GPT-4.1 and re-rank with GPT-4.1, you have built a machine that rewards itself.
The counter-evidence is substantial. UMBRELA, the open-source reproduction of Microsoft Bing's relevance assessor, reported LLM judgments correlating highly with rankings from effective multi-stage retrieval systems across five years of TREC Deep Learning tracks. And the large-scale follow-up on the TREC 2024 RAG Track — co-authored by Soboroff and NIST's Hoa Trang Dang alongside the UMBRELA team — found run-level Kendall's tau of 0.890 on nDCG@20, 0.944 on nDCG@100 and 0.929 on Recall@100 versus fully manual assessment. Both positions can hold: LLM labels are not a substitute for a human-built test collection, and they are adequate for ranking runs against each other.
The bank's own honesty is the part to copy. Its correlation with gold rankings ranged from ρ=0.69 to 0.95 depending on dataset, with TREC-COVID weakest — its qrels are dense expert biomedical judgments, the hardest thing for a general-purpose judge to reproduce. A cross-judge check against Claude Sonnet 4.6 found 27% item-level disagreement, yet system rankings still correlated at ρ=0.88–0.92. And the scope is explicitly single-stage retrievers: no reranking pipelines were evaluated, so if your stack ends with a cross-encoder, this paper does not tell you what happens.
What to Do About Your Own Retriever
This Week:
- Write down where your evaluation queries come from. Not how many — where. The bank used four distinct sources because a benchmark built entirely from generated queries measures your generator. If every query in your eval set came from the same place, that is the first defect to fix.
- Price your pool before arguing about it. Queries × candidate systems × top-100 depth, times roughly $0.001. For most enterprise search deployments that lands in three figures, which ends the budget conversation before it starts.
- Find out whether anyone has ever judged a single query-document pair on your corpus. "We watched the demo" is not a baseline, and neither is a vendor's benchmark on somebody else's data.
This Month:
- Build the pool as a keyed store, not a spreadsheet. Key on (query, document, judge model version). The model version field is what lets you re-judge later without losing the audit trail — the same lesson as prompt techniques that quietly stop paying after a model upgrade and silent post-training swaps that invalidate an eval you thought was stable.
- Run the query-formulation tests before the vendor bake-off. Instruction framing and query expansion are free to change and cost the bank more than the gap between its top five configurations.
- Put a resolution floor in your evaluation policy, in writing. Anything inside 0.001 nDCG is a tie, broken on latency and unit cost. Then hold vendors to it.
Before Your Next Renewal:
- Ask your search or RAG vendor two questions: what test collection produced your benchmark, and which model generated the labels. If the judge shares a family with their reranker, discount the number to zero. Your eval tooling — Braintrust, Langfuse or whatever you have standardised on — should be storing that provenance, and the gate you actually pay for is regression, not scoring.
- Re-run the pool against the incumbent. The marginal cost is only the documents your challenger surfaces that the incumbent never did. If the incumbent wins, you have bought a defensible renewal. If it loses by more than 0.001, you have bought a negotiating position.
The Bottom Line
Regression test suites went through exactly this arc. In 2005 they were a luxury you wrote once and let rot, because re-running them was expensive enough to need a meeting. Continuous integration made re-running nearly free, and within a decade a codebase without one was considered negligent. Nothing about testing got better; the cost of repeating it collapsed, and the standard moved.
Retrieval evaluation is at that inflection now. A bank compared 62 configurations for the price of a team lunch, found its top five configurations within 0.007 MAP of each other, and shipped the cheaper one with evidence. That is not an argument for switching embedding models. Plenty of teams will run this and confirm the incumbent — and a measured result beats a stated target every time. It is an argument for never again saying you cannot afford to check.
The evaluation was never the expensive part. Throwing it away afterwards was.
Continue Reading
- RAG Build vs Buy: Buy the Index. Build the Eval Set.
- What RAG Actually Costs: $1,308 a Month at 10M Tokens/Day
- Pinecone vs Weaviate vs pgvector: Stay on Postgres
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- DeepSeek Swapped the Model. Your Eval Didn't Notice.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
- Best RAG Platforms for Regulated Industries: Permissions First