Why a cohort of mediocre models can out-reason one great one training on itself

Reasoning RL has a supply problem. The best gains still come from training against verifiable reward — a checker that knows the right answer to a math problem or a unit test that passes or fails — and that kind of ground truth gets scarcer exactly as models get better, because you eventually run out of problems humans can reliably grade. Self-rewarding RL was the proposed workaround: let the model score its own completions and train against that. It works for a while, then it doesn't. A model grading itself has no external signal pulling it away from its own blind spots, so the errors it already makes get reinforced instead of corrected, responses homogenize, and training collapses into a narrower and often worse policy than the one you started with.

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL, from Yunhao Yang, Yuexin Bian, and collaborators, targets that specific failure mode. The fix isn't a better self-reward function — it's removing the "self" from self-rewarding. Train several models at once, none of which share parameters, and have each one score its peers' outputs instead of its own. No ground-truth labels anywhere in the loop.

The mechanism: reward from outside your own weights

The setup is a cohort of decoupled models — different architectures, different sizes, sometimes trained on rephrased versions of the same data — optimized simultaneously with RL. Each model generates responses to the same prompts; the reward for any one model's response comes from how its peers evaluate it, not from itself. That single change breaks the feedback loop that causes collapse in single-model self-rewarding RL: if the model generating the response and the model grading it are the same network, every systematic error the generator has, the grader has too, so the error gets stamped as correct and reinforced. Swap in a peer with a different architecture or training data, and the odds that both models share the same blind spot at the same moment drop sharply.

That's the paper's central empirical claim, and it's a diversity argument, not just a multi-agent one: the benefit doesn't come from having more than one model, it comes from the models being different from each other in ways that make their errors uncorrelated. The paper tests this directly — varying model family, model size, and using rephrased training samples across cohort members — and finds that more heterogeneity produces less collapse and better reasoning gains. A cohort of near-identical models would reproduce roughly the same failure mode as a single self-rewarding model, just distributed across a few copies.

What it actually buys you

The results are reported as average gains of 3.0–8.6% across seven text-only benchmarks for LLMs, and 2.3–7.2% across four multimodal benchmarks for VLMs, consistently ahead of both the base models and prior label-free methods — and the paper reports Co-RL matching or surpassing supervised approaches on some of these, despite using zero ground-truth labels anywhere in training. Two things stand out in that result set. First, it holds across both text-only and multimodal domains, which suggests the mechanism — decorrelated peer reward preventing collapse — isn't specific to one modality or task type. Second, closing the gap to supervised methods without labels is the actual point of the paper: this is a proposal for how to keep improving reasoning once you've exhausted the verifiable-reward problems that are cheap to grade.

Where I'd push on this

The mechanism is intuitively right — ensembles decorrelate errors, that's not new — but multi-agent RL papers have a history of results that look strong on the benchmark suite in the paper and get harder to reproduce once someone tries a cohort composition the authors didn't test. The open question I'd want answered before betting a training pipeline on this: how sensitive are the gains to cohort composition? "Heterogeneous model families and sizes" is doing a lot of work in that sentence, and the paper's own ablations are the evidence that diversity matters — which also means a cohort that's diverse in the wrong ways (same architecture family, slightly different scale) might not get you much of the benefit. There's also a training-cost question the abstract doesn't address: running several models through RL simultaneously is a real multiple of the compute a single self-rewarding run costs, and the paper doesn't frame the gains as compute-normalized against that baseline.

Practically, the idea is worth taking seriously if you're building reasoning models past the point where verifiable reward is available for the tasks you care about — which is an increasing share of frontier reasoning work. The strategic point isn't "use exactly this recipe." It's that diversity in the reward-generating population is a lever separate from the labeling problem: it doesn't get you around the general difficulty of grading hard reasoning without ground truth, but it does address the specific way self-rewarding RL fails, which is that a single model's self-assessment degrades in a self-reinforcing way over training. Code is released alongside the paper, which is the right test of whether the effect generalizes — I'd want to see someone run Co-RL on a cohort composition of their own choosing, on a task outside the paper's seven text and four multimodal benchmarks, before treating the specific percentage gains as a number to plan around.