RAG Benchmark Results: 297 Cases, Six Suites, Every Number

Results by suite

Ordered best to worst, deliberately. A benchmark page sorted to put the good numbers first is a marketing page.

The two numbers that look bad

These are the useful part of the page. A vendor showing you 100% on three suites has told you which three suites to ignore.

Legal (CUAD v1): 49% pass

CUAD is deliberately adversarial. Many of its cases ask about clause categories that are not present in the contract being queried, and the correct behaviour is to abstain rather than to produce something plausible. A system that scored highly here by always answering would be a worse system, not a better one.

The 49% reflects a grounding policy tuned to refuse under uncertainty. That is the right bias for legal document review, where a confident wrong answer is far more expensive than “not found in this document”. Note the column that does not move: 0 hallucinated citations, on the suite designed to provoke them.

FinanceBench: 63% pass against 96% retrieval

The gap is the finding. The right document was in context for roughly 52 of the 55 non-passing cases - so retrieval worked and the system then declined to commit.

This is over-abstention, and it is a tuning decision rather than a defect. The grounding penalty favours legal safety over financial answering, which is correct for the document-review workload the system exists to serve and wrong for a financial QA benchmark. Tuning to raise this number would make the production system worse, which is why it has not been done - and why the number is published as it stands.

Methodology

A case passes only if all three hold: the correct source was retrieved, the substantive answer was right, and no citation was fabricated. Partial credit is not given, because in document review a partly-right answer with a real-looking citation is the failure mode that costs money.

Scoring runs on five axes:

1. Retrieval hit rate — was the right source in context
2. Abstention accuracy — did it refuse when it should have
3. Citation hallucination rate — held to zero, not minimised
4. Calibration — ECE, Brier score, AUROC
5. Reranker operational counters

Reporting retrieval separately from pass rate is what makes the FinanceBench result legible at all. One combined score would have shown 63% and pointed remediation at retrieval, which was working fine. The reasoning behind that split is in building an evaluation set.

Configuration

• Reranker: Cohere rerank-3.5 (enterprise tier), direct API
• Embeddings: OpenAI text-embedding-3-small
• Judge model: Claude, injected for testability
• Store: Supabase tenant + Pinecone namespace eval-financebench
• Corpus: 31,827 vectors across 84 FinanceBench filings from 32 companies
• Full suite runtime: 3,037 seconds (~51 minutes)
• Cost per full run: ~$3 of LLM traffic, plus $0.05–$0.10 for the reranker arm

What “published” means here, precisely

This matters more than the headline number, and it is where most vendor benchmark pages are quietly misleading.

The benchmarks are public. CUAD, FinanceBench, openFDA and SEC EDGAR are all publicly available, so anyone can run them against their own system and compare like for like. That is the comparison worth having when you are evaluating a vendor.

This specific run is not independently re-executable. FinanceBench question-answer goldens are CC-BY-NC and not redistributable, and the evaluation harness lives in a private repository. So the accurate description is published and itemised, not re-runnable by you. Anyone claiming the latter about their own benchmark, without addressing golden-set licensing, is worth a follow-up question.

Limitations

All six, as recorded in the methodology document rather than summarised away:

1. CUAD is deliberately hard. Its low pass rate reflects an abstention test, not a retrieval failure.
2. FinanceBench over-abstention. A documented trade-off favouring legal safety, not a bug.
3. PDF extraction limits. The run used pdf-parse, which flattens table structure into space-separated runs. Production tenants get multimodal parsing, so these results understate production performance on tabular documents.
4. Reranker coverage variance. The Cohere arm reranked ~103 cases against Voyage’s ~26, so per-suite deltas are a configuration result rather than a proven per-call quality ranking between rerankers.
5. Not re-executable from the public repo. See above.
6. One run, one date, one configuration. 27 April 2026. Not a longitudinal study.

What this does not prove

It does not prove the system will perform this way on your corpus - benchmark documents are not your documents, and the honest way to find out is a scoped pilot on real files. It does not measure latency, cost at scale, or multi-tenant behaviour under load.

What it does establish is a methodology, a set of numbers that includes the bad ones, and a citation-hallucination count of zero across 297 adversarial cases. That last one is the claim most worth checking in any vendor you are evaluating, including this one.

Verify it yourself

• BENCHMARKS.md — full methodology and per-suite breakdown
• Raw run artefacts — the outputs behind these tables
• FinanceBench and CUAD — the public benchmarks, so you can run them against anything else
• The rest of the verification dossier, and the system these numbers come from

Frequently asked questions

What do these results measure?

Whether the system answered correctly when the answer was in the corpus, and refused correctly when it was not. A case passes only if the right source was retrieved, the answer was substantively correct, and no citation was fabricated. Retrieval hit rate is reported separately from pass rate, because the gap between them is where the interesting failures live.

Why is the CUAD legal pass rate only 49%?

CUAD is deliberately adversarial: many cases ask about clause categories not present in the contract, where the correct behaviour is abstention. A system scoring highly by always answering would be worse. The 49% reflects a grounding policy tuned to refuse under uncertainty, and it is published because a page showing only the flattering suites tells you nothing.

Why is FinanceBench retrieval 96% but the pass rate 63%?

Over-abstention, and a deliberate trade-off. The right document was in context for roughly 52 of the 55 non-passes, so the system found the answer and declined to commit. The grounding penalty favours legal safety over financial answering - correct for the document-review workload it was built for, wrong for a financial QA benchmark. Tuning for the benchmark would make the production system worse.

Can these runs be reproduced?

The benchmarks are public and the results are published, but this specific run cannot be re-executed from the public repository: FinanceBench goldens are CC-BY-NC and not redistributable, and the harness is in a private repo. So: published and itemised, not independently re-runnable. Anyone can run the same public benchmarks against their own system and compare.

What do these results not prove?

That the system will perform this way on your corpus, since benchmark documents are not your documents. They do not measure latency, cost at scale or multi-tenant behaviour under load, and the FinanceBench figures assume basic PDF extraction rather than the multimodal parsing production tenants receive. They are one run, dated 27 April 2026, on one configuration.