RAG techniques don't add up, and the winner changes with corpus size

Most RAG improvements I read about are measured the same way: take a baseline pipeline, swap one stage, report the delta. A better reranker, a smarter chunker, a query rewriter, an agentic verification loop. It is natural to assume that if you adopt three of them you get something close to the sum.

A new paper from IBM and the Weizmann Institute, Retrieval Augmented Generation Compositionality by Ido Levy, Segev Shlomov and Leshem Choshen, tests that assumption directly. Their answer is that it mostly does not hold for retrieval, roughly holds for faithfulness, and the ranking of complete pipelines shifts as the corpus grows. None of this will shock anyone who has tuned a production RAG system. What the paper adds is a careful measurement of how large the effects are.

How the study is set up

The authors built an open-source framework, lasagna, that splits a RAG pipeline into eight stages behind fixed, typed interfaces: embedding, chunking, index-time enrichment, retrieval, query augmentation, reranking, context compression, and agentic quality gates. Isolation tests check that toggling one stage does not silently change another, so a measured difference belongs to the technique rather than to incidental implementation changes.

The full factorial over those eight dimensions would be 30,720 configurations. The authors planned a 242-run D-optimal subset sized to keep all main effects and all 28 pairwise interactions estimable, but executed 167 of those runs: 63 GraphRAG-enriched runs were dropped after an ablation showed no NDCG@10 change, and 12 timed out. That gap matters. Estimating all 28 interactions jointly requires 232 parameters, more than the executed design can support, so the headline composition result is a comparison of predicted versus realized gains rather than a full interaction model.

Around the factorial sit 138 targeted ablation configurations and a corpus-scaling sweep from 5K to 200K documents, 450 configurations in total over eight benchmarks merged into one multi-domain corpus. Hypotheses were declared and hashed before execution, and the whole study cost about $5,200 in compute.

Retrieval gains are sub-additive

The three largest single-component retrieval improvements (cross-encoder reranking, disabling MMR, and document-awareness prefixes) predict a +46.6% NDCG@10 lift over the default pipeline. The pipeline that combines them delivers +17.2%, a 63% shortfall. Adding all five individually positive retrieval techniques widens it to 72%, and across 17 tested combinations the shortfall ranges from 7% to 72%, with a median of 41%. The baseline was realistic: 0.412 NDCG@10, against 0.403 for BM25 alone and 0.436 for a recommended off-the-shelf dense encoder.

Faithfulness behaves differently. A one-factor-at-a-time estimate predicts +28%; the best full pipeline realizes +32%.

The paper's explanation is functional overlap. Reranking already does some of the filtering you gain by turning off MMR, so the two compete for the same headroom. The two retrieval interactions that survive multiple-comparison correction both involve the embedder: Embedding x Enrichment and Embedding x Reranking. A strong embedder leaves less for a reranker to fix. Faithfulness levers sit at different stages (query augmentation changes what is retrieved, agentic gating filters it before generation), so they duplicate less of each other's work.

In physics terms, this is the difference between independent perturbations and coupled ones. You can add first-order corrections only when the cross terms are small. For retrieval they are not.

The two metrics barely talk to each other

Across matched configurations, NDCG@10 and RAGAS faithfulness have a pooled Pearson correlation of -0.01. Retrieval quality is driven mostly by embedding, enrichment and reranking; faithfulness mostly by query augmentation and agentic gating. A pipeline tuned on retrieval metrics alone can retrieve well and still produce weakly grounded answers, and the reverse holds too. If your dashboard shows one of the two, you are optimizing a projection of the system.

The complete profiles make the tradeoff concrete, with one caveat up front: each profile's faithfulness was scored using its own deployment-native answer prompt, not a shared one. The profile differences below therefore mix pipeline and prompt effects, and should not be read as component effects alone. On a held-out QA mixture, standard leads NDCG@10 (0.860) at 5.3 s mean latency, with faithfulness of 0.739. The agentic balanced and max-reliable profiles push faithfulness above 0.91 but take roughly 12 to 14 seconds. speed answers in 0.7 s and gives up a lot of quality. No profile wins everything, and max-reliable, which enables the most features, gains only 0.005 faithfulness over balanced while losing retrieval quality and latency.

The best pipeline at 5K is not the best at 200K

Most published RAG ablations run on 5K to 50K documents. The authors scaled their base corpus up to 200K with distractors drawn from the same domains. standard and speed degrade log-linearly in NDCG@10; balanced and max-reliable stay roughly flat. standard leads at 5K and 15K, balanced leads from 25K on, and at 200K balanced is ahead by 13.7% relative.

Faithfulness scales differently again: it peaks around 50K, dips at 100K and recovers by 200K. The authors hypothesize that a larger corpus adds paraphrased or overlapping evidence, so an answer can stay grounded even when the gold passage drops out of the top-k. If you benchmark on a sample of your corpus and ship the winner on the full index, this is the result to take seriously.

Smaller findings worth noting

• Chunker choice alone spans 0.126 NDCG@10 across 20 configurations, a 36% relative difference that is larger than most factorial main effects.
• The 2019 MS-MARCO MiniLM-L6 cross-encoder outranked all four recent rerankers tested on these domains.
• Matryoshka truncation of BGE-M3 to 768 dimensions cut vector storage and search cost by 25% and improved NDCG@10 by 2.3%.
• Hallucination and answer-quality gates lowered faithfulness. In one multi-hop example, gate-triggered refinement removed 14 of 15 supporting chunks.

Caveats

Beyond the reduced factorial and the prompt difference above, the headline comparisons rest on one constructed English test bed with at most 93 retrieval and 200 generation queries per experiment. Faithfulness is LLM-judged; a second judge preserved rankings but shifted absolute scores, and a third showed only moderate agreement (Cohen's kappa 0.35). The pre-declared interaction threshold for faithfulness (H1) was not met: the interaction is real (p = 0.007) but small. The profiles are candidates to validate, not recipes.

What I take from it

The core idea is intuitive: components that address the same failure mode overlap, and the corpus you test on shapes the answer. The value here is in the magnitudes. A 41% median shortfall on stacked retrieval gains is large enough to change how I read any single-component RAG result. In practice that means three things: evaluate a new component inside a complete pipeline, report retrieval and faithfulness together with latency, and rerun the comparison at the corpus size you will actually serve. The lasagna framework makes the first of those a one-line registration, which removes most of the excuse for not doing it.