Why your retriever's leaderboard score might be lying to you

The gap nobody was measuring

Most retrieval benchmarks fail agentic RAG in one of two ways. Either the corpus is huge and the query set is small — a handful of hand-picked queries against billions of documents — or the query set is large but the corpus is trimmed down to a few million documents so evaluation stays cheap. Neither setup looks like what a production agent actually does: search a web-scale corpus with a query that an LLM just rewrote from a user's messy, conversational ask.

That mismatch is the starting point for Q2D-Web (Query2Doc-Web), a benchmark from researchers including Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang, and Denis Bykov. I've spent enough time tuning first-stage retrievers for production RAG pipelines to recognize the failure mode they're describing: a retriever looks great on a public benchmark, then underperforms once you swap in the query reformulations your agent actually produces, because those reformulations don't look like human search queries. They're longer, more explicit, sometimes multi-intent, and shaped by whatever conversation thread preceded them.

What's actually in the benchmark

Q2D-Web pairs a 190-million-document web corpus with 70,000 agentic search queries across ten languages, all reformulated from real user queries and their conversation threads in production systems. That's the part that matters most: these aren't synthetic queries generated by prompting an LLM to "imagine a search query." They're reformulations an agent actually issued, sampled from real usage.

The harder problem with a benchmark this size is relevance judgment. You can't manually label every document against every query at 190M scale. The authors' answer is to build three separate labeled sets and check whether they agree:

• Agent citations — documents the production agent actually cited when answering.
• Production rankings — signal derived from how the live retrieval system ranked results.
• Combined — a union of both, extended with LLM-based judgments over pooled-but-unlabeled documents specifically to catch false negatives that citation and ranking signals alone would miss.

They then benchmark 13 retrievers spanning lexical (BM25-style), dense embedding, and late-interaction architectures against all three judgment sets. The headline finding: a retriever's relative ranking is largely insensitive to which judgment set you use — a model that wins under agent citations tends to win under the combined set too. What isn't stable is performance across topical domains, query languages, and query types. A retriever that leads on English factual queries can slide on multi-hop or non-English ones. If you're picking a retriever for a multilingual agentic system, the aggregate leaderboard number is close to useless; you need the breakdown.

The subsampling trick

The part of this paper I'd actually put into practice is the corpus subsampling result. Evaluating 13 retrievers against 70k queries over 190M documents is expensive — every retriever has to score against the full corpus for every query. The authors test whether you can shrink the corpus and still trust the results.

Their approach: run reciprocal rank fusion (RRF) over the pooled results of all the retrievers being evaluated, then keep only the top-ranked third of the corpus by that fused score — about 63 million documents instead of 190 million. Evaluating on this subcorpus preserves the full-corpus model ranking under the combined judgment set, while Recall@1000 only rises by 3 to 7 points versus the full run.

That's a genuinely useful engineering result, not just a benchmark curiosity. If you're iterating on a retriever — swapping embedding models, tuning a reranker cutoff, testing a new chunking strategy — you don't need the full corpus for every iteration. Pool your candidate runs, fuse them with RRF, keep a third of the corpus, and you get a cheap proxy that tracks the expensive ground truth. The absolute recall numbers shift slightly (predictably upward, since you've discarded some of the hardest true negatives), but the thing you actually care about during iteration — which model is better — doesn't change.

Why this matters beyond the leaderboard

The deeper point Q2D-Web makes is about evaluation methodology, not just retriever performance. If your eval set is human-written queries against a small corpus, you're measuring something adjacent to your production problem, not your production problem. Agentic RAG systems increasingly sit between a user and a retriever with an LLM doing query rewriting in between, and that rewriting step changes the query distribution enough that retriever rankings measured on human queries don't transfer cleanly.

For anyone building or evaluating retrieval for an agentic system, two things from this paper are worth adopting directly: evaluate against reformulated queries that resemble what your agent actually emits, not raw user text, and use a corpus large enough that a retriever's recall ceiling isn't artificially inflated by having too few documents to confuse it. The subcorpus sampling trick is the practical bridge that makes the second point affordable. A public leaderboard is available alongside the benchmark, linked from the paper, for anyone who wants to see where the 13 evaluated retrievers land before running their own.