Coding agents fail scientific software in ways SWE-bench never measured

A new benchmark called SWE-bench Science just gave me a number I've been missing for a while: how well do coding agents actually do on scientific software, as opposed to the web apps and CLI tools that populate most SWE-bench variants? The answer, from the paper's own headline result, is that the best agent tested — Claude Code running Opus-5 in max mode — resolves fewer than half its tasks. That's not a knock on the agent. It's a signal about what kind of failure scientific code repair actually is, and it matches what I see running agentic coding against research codebases day to day.

Why scientific repos are a different benchmark

Most coding-agent benchmarks measure whether a patch makes a failing test pass. That's a reasonable proxy for a web framework or a CLI utility, where the test suite is close to a complete specification of correct behavior. Scientific software doesn't work that way. A numerical solver, a signal-processing pipeline, or a genomics tool can pass its unit tests while quietly producing a result that's numerically wrong, unstable at the boundary, or inconsistent with the physics or biology the code is supposed to model. The authors put it plainly in their abstract: failures in scientific code can compromise "not only program behavior but also the evidence underlying scientific conclusions." A green checkmark and a correct scientific result are not the same claim.

SWE-bench Science is built around that gap. It's a repository-level benchmark of 119 tasks pulled from 98 real GitHub repositories across 20 scientific domains, and each task is labeled with one of three paradigms: issue-driven (a bug report or GitHub issue points at the problem), expert-exploratory (the agent has to find the issue without a clear pointer, the way a domain expert would), and engineering-integration (the fix has to thread through multiple parts of a working system, not just patch one function). That range matters — it's the difference between fixing a typo in a well-specified function and correctly modifying a piece of a larger scientific pipeline where the fix has to stay consistent with everything downstream.

The headline number, and why it's the interesting part

Across all three paradigms, the strongest agent in the study — Claude Code with Opus-5 (max) — comes in under 50% pass@1. Every other configuration does worse. This is the frontier model, in its most capable configuration, failing the majority of real repair tasks on real scientific repositories. I don't read that as an indictment of the model; general-purpose SWE-bench numbers for the same class of agents are considerably higher. I read it as evidence that scientific software repair is a harder distribution than the benchmarks we've been using to say agents are "good at coding."

The paper backs that up with a taxonomy of four recurring failure mechanisms, and I found all four recognizable from firsthand agent-in-production debugging:

1. Deficits in scientific knowledge or abstraction — the agent doesn't know the domain well enough to recognize what "correct" means for this code.
2. Misguided exploration or surface-level repair — it patches the symptom (the line that threw the error) rather than the actual defect.
3. Incomplete repair coverage or system integration — the fix works locally but doesn't propagate through the rest of the pipeline.
4. Failure to generalize scientific knowledge beyond observed cases — the agent can apply a pattern it's seen before but can't extend the underlying principle to a new but related situation.

None of these are exotic. They're the same failure modes I'd expect from a junior engineer parachuted into an unfamiliar scientific codebase without a domain expert to check their work — except the agent doesn't know it needs the check.

The ablation: guidance is not automatically good

The part of this paper I think matters most for anyone actually deploying coding agents is the controlled ablation. The authors ran a paired comparison where they stripped out explicit scientific guidance from the agent's context while keeping the repository and executable engineering context intact — same repo, same test harness, same task, just with or without domain hints layered in.

The result cuts against a comfortable assumption: that more domain context is always a free win. It isn't. Well-grounded, correctly aligned scientific guidance narrows the agent's search space, improves average pass rates, and cuts token usage — it acts the way a good code review comment should, ruling out wrong turns early. But guidance that's poorly aligned with the actual defect does the opposite: it anchors the agent on the wrong hypothesis, and the paper is explicit that it "does not necessarily improve exact repair success" even when it superficially looks like domain expertise is being applied. An agent primed with confident-sounding but off-target scientific framing will chase that framing instead of the bug.

That's a genuinely practical finding, not just an academic one. Every team building coding agents against specialized codebases — scientific, financial, regulatory, whatever the domain — eventually asks whether to inject domain context into the system prompt or retrieval layer to help the agent. The instinct is that more relevant context is strictly better. SWE-bench Science's ablation says the value of that context is conditional on its alignment with the specific defect, and a wrong or approximate hint can be worse than no hint, because the agent trusts it and stops looking elsewhere.

What I take from this

I don't build coding agents that patch scientific instrument software, but I build and operate agentic systems where a plausible-looking output can be wrong in a way that's expensive to catch late — trading logic, research pipelines, production code review. The lesson generalizes past the scientific-software framing: guidance you inject into an agent's context is a bet, not a guarantee, and its expected value depends entirely on how well it's grounded in the specific instance in front of the agent. Generic domain priming is risk, not just help. If you're going to inject domain guidance into a coding agent's context, the return depends on how confident you are that the guidance is actually right for this bug — not just plausible for the domain in general. Confident-sounding but generic context is exactly the failure mode this ablation catches, and it's worth checking your own agent harnesses for it before scientific or otherwise high-stakes code starts moving through them autonomously.