Judging a code patch without running it
Training a coding agent needs a referee to grade its patches, and the standard one spins up a Docker environment per repository to run the tests. A new paper judges correctness by exploring the repo instead — a fully environment-free SFT and RL pipeline that matches the execution-based version.
The tax that never makes the slide
Every coding agent that resolves real GitHub issues was trained inside a loop, and that loop has a referee. When the model proposes a patch, something has to decide whether the patch is right. In the standard recipe that referee is execution: you stand up the repository's environment — usually a Docker image pinned to the right dependencies — run the unit tests, and read off pass or fail. That verdict does double duty. It filters which trajectories are good enough to fine-tune on, and it becomes the reward signal for reinforcement learning.
The referee is expensive in a way that rarely shows up in a results table. Every repository needs its own environment. Dependencies rot, build steps break, and a meaningful fraction of issues simply will not containerize without hand-holding. The compute to run tests thousands of times per training run is real, but the setup and upkeep of per-repo environments is the part that quietly caps how much data you can afford to train on.
A recent paper, Dockerless: Environment-Free Program Verifier for Coding Agents, asks the question that's obvious in hindsight: what if the referee never runs the code at all?
What the verifier is actually for
It helps to be precise about where the verifier sits, because it's load-bearing twice. During supervised fine-tuning, it decides which of the agent's trajectories are correct enough to imitate — bad filtering means you train the model to reproduce plausible-looking mistakes. During reinforcement learning, it is the reward — the number the policy optimizes against, thousands of times over. Get the verifier wrong in either place and the whole pipeline drifts. This is why "how do we judge a patch" is not a side detail of training coding agents. It is close to the center of it.
Reading instead of running
Dockerless replaces execution with an agent. Given a candidate patch, it explores the repository — reading the changed code, the surrounding tests, the issue it's meant to close — and gathers evidence for whether the patch does what it should. Then it renders a verdict.
The authors are specific that this is not reference-matching. It doesn't diff the candidate against a known-good patch and score the overlap. That distinction matters, because reference-matching has a low ceiling: two correct patches can look nothing alike, and a patch can echo the reference almost exactly while fixing the wrong thing. A verifier that reasons about behavior instead of surface form is at least aiming at the right target.
Both approaches feed the same two consumers — the SFT trajectory filter and the RL reward. Dockerless swaps the per-repo environment for repository exploration and a judgment call.
The numbers
Two claims stand out. As a verifier in isolation — scored on a benchmark of patches labeled correct or incorrect — Dockerless beats the strongest open-source verifier by 14.3 AUC points. That's the raw quality of the judge, before it touches any training.
The more consequential claim is the full pipeline. With Dockerless choosing the SFT trajectories and supplying the RL reward, and no execution anywhere, the resulting model resolves 62.0% of SWE-bench Verified, 50.0% of Multilingual, and 35.2% of Pro — ahead of the Qwen3.5-9B baseline by 2.4, 8.7, and 2.9 points, and level with an environment-based pipeline that actually ran the tests.
The environment-free pipeline lands even with an execution-based one, with its widest margin on Multilingual — where per-repo containerization is the biggest headache.
The Multilingual gain is the one I'd underline. Multi-language repositories are exactly where containerization hurts most: more toolchains, more build systems, more ways to fail before a single test runs. A verifier that never builds an environment sidesteps all of it, and that's where it posts its largest margin. The mechanism and the result point the same direction, which is a good sign.
Where I'd push on this
I lead teams that ship agentic systems, and I've learned to distrust any reward that's cheaper than the thing it replaces until I understand what it gave up. A few things I'd want to see before wiring this into my own pipeline.
The first is reward hacking. When the reward is a model reading code, the policy under RL will hunt for whatever the judge over-credits — patches that read as convincing but don't actually fix the bug. Execution has one hard virtue: a failing test is a failing test, and you can't talk it out of the verdict. A learned verifier trades that hardness for coverage, and an aggregate AUC number doesn't tell me how the errors are distributed. I'd want the false-positive rate on patches that pass review but fail tests, because those are the ones that quietly poison training.
The second is where the judge's knowledge runs out. Reasoning from static evidence works when the bug is legible in the code — a wrong condition, a missing case. It's on much thinner ice for anything whose correctness only appears at runtime: a race, a numerical tolerance, a performance regression, an integration with a service you can't see by reading. Execution catches those precisely because it doesn't need the code to be readable.
The honest framing is that this isn't execution-free correctness. It's execution-free plausibility, calibrated well enough to train on. For filtering trajectories and shaping a reward, plausibility at scale can be worth more than exactness at a fraction of the volume. For a final gate on production code, I'd still run the tests.
What it changes
The binding constraint on training software agents has been environment engineering, not model architecture or raw compute. If a learned verifier can match execution-based post-training, that constraint loosens. You can train on repositories you could never afford to containerize — old code, obscure toolchains, private monorepos, anything where standing up a reproducible test environment costs more than it's worth. The price of a verdict drops from "build an environment" to "run inference," and the training distribution widens to match.
Distribution is usually what's actually scarce. If Dockerless holds up under adversarial pressure — and that's the part I'd want independent replication on — the interesting outcome isn't a cheaper version of today's pipeline. It's agents trained on the long tail of software that execution-based verification was always too expensive to reach.