UNREAL: one frozen LLM as both retriever and long-context reader

Most production systems I have worked on carry two separate answers to the same question. When the evidence fits in the prompt, we hand the whole thing to the model and hope attention finds the right paragraph. When it does not fit, we stand up a retriever, usually an embedding model plus a reranker, and feed the model whatever comes back. Two stacks, two sets of failure modes, two things to keep in sync every time the generator changes.

UNREAL: Unifying Retrieval and Long-Context with a Single Model, from Kinderman, Hoffer, Soudry, Ginsburg and colleagues at NVIDIA and Technion, argues that these are the same operation at different scales: query-conditioned selection of evidence. Their proposal is to let the generator do the selecting, using its own internal states, with a tiny trained adapter and no backbone finetuning. I think the framing is correct, and the numbers are strong enough to take seriously. Some caveats matter, though, before anyone rips out a retrieval stack.

What UNREAL actually trains

The method extends the authors' earlier INTRA work, which did intrinsic retrieval in encoder-decoder models, to the decoder-only and hybrid models people actually deploy. There are three pieces.

Chunk embeddings. Every chunk (about 140 tokens on average) is passed independently through the frozen LLM, and the residual-stream states from one intermediate layer are kept as a multi-vector representation. Middle layers beat final layers here, consistent with prior work on layer-wise semantics.

Residual-state queries. The query is built as [soft prompt, BM25 top chunks, question, R retrieval tokens]. A small learned soft prompt at the start of the sequence puts the model into a retrieval mode. The retrieval tokens are also learned vectors, placed last, so under the causal mask their states see both the question and the BM25 context. Their states are read out from several layers, mixed with learned weights, and scored against each chunk with ColBERT-style MaxSim late interaction.

Generation. The top-n chunks are re-encoded as plain text in front of the question, and the same frozen model answers.

The trained parameters are the leading soft-prompt vectors, the retrieval-token embeddings and the layer-mixing coefficients, trained with a multi-positive InfoNCE loss over oracle chunks and hard negatives. That comes to fewer than 500K parameters, under 0.005% of each backbone. Because the read-out sits on the residual stream, it works the same way on dense attention, linear-attention hybrids and Mamba-attention hybrids.

The corpus-scale results

On a 3B-token, 21M-chunk Wikipedia 2018 index, the metric is complete-evidence recall@10: a multi-hop question counts only if every oracle chunk shows up in the top ten. That is a demanding bar, and it is the right one for multi-hop work, where getting one of two passages buys you nothing.

All four backbones (two dense, a linear-attention hybrid, and a Mamba-attention hybrid) beat BM25, BGE, Qwen3-Embedding, a LateOn multi-vector retriever, hybrid RRF, and embedding-plus-Jina-reranker pipelines on most of the eight datasets. The best model moves HotpotQA from 49.1% to 73.2% and 2WikiMultiHopQA from 31.7% to 60.1%. MuSiQue goes from 8.8% to 14.4%, which is still low, and that is a useful reminder of how hard genuine compositional retrieval remains.

The end-to-end numbers matter more to me than recall. With Nemotron-3.5-Lightning fixed as the generator and only the top-5 context varied, UNREAL reaches 52.6 EM on HotpotQA against 43.3 for the best retriever-plus-reranker baseline, with similar margins on 2Wiki and MuSiQue. Better recall turning into better answers is the part that is easy to lose in practice, and here it holds.

The same selector, inside one prompt

The second half of the paper takes the exact same trained module, with no long-context training data, and points it at a single long prompt: chunk it, rank the chunks, keep the top n, and generate from those alone. On NoLiMa, a needle benchmark built to have no keyword overlap between question and needle, full-context accuracy at 128K tokens is 1.0% and UNREAL reaches 24.83%. LV-Eval F1 at 256K goes from 49.97 to 54.66. On HELMET's RAG subset it beats full-context inference for both generators tested.

Two findings here match what I see in production. First, accuracy against n follows an inverted U: more chunks raise recall until the added distractors start to hurt generation. A bigger window gets the evidence into the prompt; it does not make the model use it. Second, the cost crossover is real. Chunks are encoded independently and generation attends only to the selected text, so cost grows roughly linearly with context. The authors put the FLOP break-even near 18K tokens and measured time-to-first-token wins on an H100 from about 32K onward.

Where I would push on it

The question I care about is simple: do the gains hold under fair baselines? Mostly yes, with three things I would want answered before relying on it.

BM25 is load-bearing. The query is conditioned on the BM25 top-5. The ablation shows that removing it drops HotpotQA complete-evidence recall@10 from 72.93 to 56.73. That is still above the 49.1 of the best baseline, which is reassuring, but a meaningful share of the gain comes from a cheap first-stage retriever feeding the query. The system is not BM25-free, and it should not be described that way.

Training and evaluation share a domain. The baselines are general-purpose retrievers; UNREAL is contrastively trained on Wikipedia QA and evaluated on Wikipedia QA. The authors say plainly that transfer to other domains and languages is future work. Enterprise documents, code and tables are where I would test first, and where a domain-tuned embedding model is the fairer comparison.

Index cost. Every chunk stores several pooled vectors from a large model rather than one small embedding. Recall rises monotonically with the number of vectors per chunk, so storage trades directly against quality. Re-indexing a large corpus whenever the backbone changes is a real cost, although teams running their own retriever already pay a version of it.

The authors also scope the long-context claim to sparse-evidence tasks. Summarization or anything that needs most of the input is outside it, and their call for benchmarks that separate evidence regimes is the right one.

Why this matters for production

The appeal is not the leaderboard. It is the operational simplification: one model that encodes, ranks and answers, so the retriever cannot drift out of alignment with the generator, and a long prompt and a corpus go through the same code path. An adapter under 500K parameters is cheap enough to train per backbone and per domain. If the in-domain results transfer even partially, I would use this as the selection stage underneath an agentic retrieval loop, which is where the authors position it too.

The broader point I take from the paper is the one in their discussion: models should be trained to recall, and long-context evaluations should report evidence recall next to answer accuracy. Treating a larger context window as the solution has always mixed those two together.