Turning open-ended tasks into a game so the reward can't lie

The problem RLVR never solved

Reinforcement Learning with Verifiable Rewards is the reason reasoning models got so much better so fast. Feed a model a math problem or a coding task, check the answer against ground truth, and you have a reward signal that cannot be gamed, cannot drift, and costs almost nothing to compute. That's the whole trick — verification is what makes large-scale RL tractable.

The trick stops working the moment the task doesn't have a checkable answer. Summarization, creative writing, dialogue, anything where "good" is a judgment call rather than a fact — RLVR has nothing to grab onto. The field's answer has been to bolt on a substitute: a human preference model, a learned reward model, or increasingly an LLM acting as judge. All three carry the same tax. Judges have their own biases and blind spots, they're expensive to query at RL scale, and their competence caps what they can reliably evaluate. You end up training a model against another model's opinion, which is a shakier foundation than it looks once you're doing it millions of times.

A new paper, From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement (Wang et al., COLM 2026), goes after this from an unusual angle. Instead of building a better judge, it removes the judge.

The self-supervised move: transform the task, not the reward

The paper's framing borrows a page from self-supervised learning. You don't need labels if you can construct a pretext task where the data supplies its own supervision — predict the masked word, reorder the shuffled patches, whatever. The correctness signal comes from the structure of the task itself, not from an external annotator.

RLSVR (Reinforcement Learning with Self-Verifiable Rewards) applies that same move to RL reward design. Rather than asking "how do we verify a summary is good," the authors ask "can we transform the act of writing a summary into something that has a built-in, checkable outcome?" If yes, you get RLVR's cheap, unbiased reward on a task that was previously off-limits to it.

SpyRL: turning the task into "Who Is the Spy?"

Their concrete instantiation is SpyRL, modeled on the social deduction game Who Is the Spy?. The setup, applied to something like summarization:

• A group of agents each receive the same underlying task (summarize this article, write this story) but with asymmetric information — one agent (the spy) is working from a subtly different brief or source than the rest.
• All agents produce their output.
• The agents then vote on who among them is the spy, using only the outputs as evidence.

Because the organizers know which agent was assigned the spy role, the correct vote outcome is known ahead of time — the reward is verifiable by construction, exactly like checking a math answer, with zero judge model in the loop. What makes this more than a party trick is the coupling: to correctly spot the spy, an agent has to actually read and reason about output quality and consistency, and to avoid being caught as the spy, the spy has to produce an output good enough to blend in. Voting accuracy is entangled with output quality, so optimizing the verifiable voting reward pushes the policy toward better summaries and better writing as a side effect — without ever scoring a summary directly.

That's the core insight worth sitting with: the reward doesn't grade the essay. It grades whether the group can find the odd one out, and being hard to catch (or good at catching) turns out to require being good at the underlying task.

What the experiments show

The authors test SpyRL on text summarization, creative writing, and — as a sanity check — mathematical reasoning, which already has verifiable rewards available. Two results stand out:

1. On the genuinely open-ended tasks (summarization, creative writing), SpyRL beats existing self-improvement baselines that rely on LLM judges or preference models. The self-play voting signal turns out to be a better training target than a judge score, not just a cheaper one.
2. On math, where RLVR already works, SpyRL still yields consistent gains layered on top. That's a useful control — it suggests the social-deduction framing isn't just a workaround for missing verifiers, it's picking up a training signal (something like calibrated skepticism toward subtly wrong or off-brief content) that transfers even where ground-truth verification already exists.

Taken together, the results support the paper's central claim: task transformation is a general recipe for extending RLVR-style self-improvement past math and code, not a one-off hack specific to summarization.

Why this matters if you're building with RL post-training

I've spent enough time on the production side of RL fine-tuning to know that the LLM-judge tax is real. Every judge call is inference cost, every judge is a model with its own quirks and drift, and every RL run built on judge scores inherits whatever biases that judge has baked in — position bias, length bias, style preferences that have nothing to do with actual quality. Teams work around this with judge ensembles, calibration passes, human spot-checks — all real work that scales linearly with how much you're training.

What RLSVR offers isn't a better judge, it's a structural argument for why you might not need one: if you can find a game-like transformation where the correct outcome is knowable in advance and where winning the game requires being good at the underlying task, the verification problem disappears rather than getting solved. That's a narrower condition than it sounds — it requires a task where asymmetric information and adversarial concealment naturally apply, and summarization and creative writing are relatively forgiving cases. Whether this generalizes to, say, code review quality or open-ended agentic tool use is an open question the paper doesn't claim to answer. But as a design pattern — don't build a better critic, build a game where the outcome referees itself — it's a genuinely useful addition to the RL post-training toolkit, and I'd expect to see more task-transformation approaches like it as teams hit the same judge-cost wall.