When the answer is a set, not a chain: set-distance rewards for radiology reports

Most of what reinforcement learning has done for language models lately rests on one convenient property: the answer is checkable. A math answer matches or it doesn't. A unit test passes or it fails. Reinforcement learning with verifiable rewards turned that property into a training signal, and it works well — as long as correctness is a point you can hit.

Radiology reports are not points. The findings section of a chest X-ray report is an unordered collection of mostly independent observations — heart size, lung fields, pleural spaces, bones — and there are many equally correct ways to phrase and order them. A new paper, SDR: Set-Distance Rewards for Radiology Report Generation by Gulluk, Van Puyvelde, Van Criekinge and Gevaert, starts from that mismatch and builds a reward that fits the shape of the output. The core idea is clean, and the second half of the paper — reusing the same signal at test time — is the part worth stealing.

The reward-shape problem

The two standard reward families in this line of work both assume structure a radiology report doesn't have. Exact-match rewards treat the reference report as the single right answer: reorder the findings or paraphrase one sentence and the reward drops to zero, even though the report is clinically identical. That makes the signal sparse and, worse, wrong — it penalizes valid outputs. Process rewards, the other family, score the intermediate steps of a reasoning chain, which presumes step k depends on step k-1. Findings in a report are orthogonal; "no pleural effusion" does not follow from "heart size is normal". There is no chain to score.

So the paper reframes the problem: stop treating a report as a sequence and treat it as a set.

Reports as point sets

The mechanics are deliberately simple. Split the generated report and the reference report into sentences. Embed every sentence with a frozen, off-the-shelf sentence transformer. Each report is now an unordered set of vectors. Compute a set-to-set distance between the generated set and the reference set, and use that as the reward for GRPO post-training.

Two properties do the work. The reward is permutation-invariant, so ordering is free — as it should be. And it is continuous, so partial credit exists: a report that nails four of five findings and paraphrases the fifth lands close to the reference instead of scoring zero. Dense, graded rewards are exactly what policy optimization wants; exact match hands it a cliff.

It is also worth stressing what stays frozen: the sentence encoder never trains. There is no learned reward model to co-adapt with the policy and no preference data to collect. The reward is a fixed geometric measurement.

Does it hold up on clinical metrics?

Across two datasets and three open vision-language models (Qwen3-VL-2B, Qwen3-VL-4B, Gemma3-4B), GRPO with set-distance rewards beats both supervised fine-tuning and exact-match GRPO on every headline metric — on average 6.8% relative on BERTScore, 7.8% on RadGraph F1, and 4.5% on CheXbert F1.

The number I watch is RadGraph F1. BERTScore is itself embedding-based, so an embedding-based reward improving it is halfway expected. RadGraph and CheXbert measure something different — extracted clinical entities, relations, and finding labels. Improvement there says the model is not just drifting toward the reward's own geometry; it is getting the findings more right.

One signal, three jobs

At inference time there is no reference report to measure against, so the authors flip the construction: score each candidate by its set distance to the embeddings of training reports, and keep the best of N samples. That reference-free selection beats random choice by an average 16.4% relative on BERTScore — and it works as a pure output-side filter on closed models too (Mistral-Small, Gemini 2.5 Flash-Lite, GPT-4o-mini), with no access to weights or logprobs.

The next step is the one I would take to production. Because the score is computed over a set of sentences, it can be evaluated while a candidate is still being generated, and low scorers can be killed mid-stream. That pruning cuts generated tokens by more than half while preserving the Findings quality of full best-of-N selection. Best-of-N is usually a blunt cost multiplier — N times the tokens for one answer. A ranker cheap enough to run mid-generation turns it into a controlled spend, and that is the difference between a benchmark trick and a deployable inference strategy.

Where I would push

Three things I would want to see before trusting this in a clinical pipeline.

The frozen encoder is the ceiling. The reward cannot distinguish anything the encoder embeds identically, and sentence embeddings are notoriously loose about negation — a serious concern in a domain where half the sentences start with "no". The CheXbert gains are reassuring, but I would want an error analysis specifically on flipped findings.

Reference-free scoring is a typicality prior. Ranking candidates by distance to the training corpus favors reports that look like average reports. A rare but correct finding sits far from the training cloud and risks being scored down — fine for aggregate metrics, exactly wrong for the cases a radiologist most needs flagged.

And continuous rewards are softer targets than exact match. Optimization will find whatever the encoder over-credits. Freezing the encoder blocks co-adaptation, but it does not eliminate reward hacking; it fixes the surface being hacked.

The general lesson

Match the reward geometry to the output geometry. Chains get process rewards, points get exact match — and sets should get set distances. A lot of production outputs are sets, not just radiology reports: extracted fields, summary bullets, action items, entity lists. I have watched teams score set-shaped tasks with sequence-shaped metrics and then wonder why RL fine-tuning stalls.

The other lesson is economic: a good reward is a reusable asset. The same frozen-encoder distance trained the model, ranked its samples, and pruned its decoding. Before commissioning a bespoke learned reward model, it is worth checking whether a frozen encoder and the right distance already give you a signal you can spend in three places. The authors say their code is public, so checking is cheap.