Repair or resample: what actually fixes multi-agent LLM failures
A new controlled-replay framework shows that most rerun-based repairs of multi-agent LLM failures are lucky resamples, not causal fixes — and that targeting the failure's actual symptom roughly triples the real repair rate.
The question nobody was checking
When a multi-agent LLM system fails a task and you rerun it, and this time it succeeds, did you fix anything? Or did you just get a different roll of the dice?
That's the question at the center of Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems, a new paper from Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, and Xiaohong Chen. It's a question I don't think most teams building agent debugging tools have actually answered, because most of us don't have the infrastructure to tell the difference between a causal repair and a lucky resample. The paper builds that infrastructure, and the numbers it produces are not flattering to the current state of the art.
Why rerun-based repair is hard to trust
Almost every practical approach to debugging a failed multi-agent trajectory today looks the same: something went wrong, so rerun the whole thing, maybe with a tweaked prompt or a different seed, and see if it comes out right. The problem is that LLM sampling is stochastic. If you rerun a multi-agent system enough times, some fraction of failures will resolve themselves through pure variance — not because anyone understood or addressed what actually broke. A repair method that can't distinguish "I fixed the root cause" from "I happened to draw a better sample this time" isn't really a repair method. It's a slot machine with a debugging UI on top.
Measuring that distinction requires being able to replay a failure under controlled conditions — same context, same upstream state, only the intervention varied — which is exactly what unconstrained rerunning can't give you.
SymTrace: replay before the anchor, regenerate after
The paper's first contribution is SymTrace, a controlled evaluation framework built to make that measurement possible. It records a full MAS execution trajectory and lets you set an intervention anchor at a specific point in that trace. During replay, SymTrace reconstructs everything before the anchor deterministically from the recorded logs, and only regenerates the trajectory downstream of it. That means when you apply a candidate repair, you know it's being tested against the exact same upstream conditions every time — the comparison is apples to apples instead of two different runs that happened to diverge somewhere upstream for unrelated reasons.
That framing directly targets the reproducibility problem: without an anchor and a deterministic prefix, you can never be sure whether a rerun "fixed" the failure or just avoided it by taking a different path from the start.
SymFail: 536 failures, labeled and graph-linked
The second contribution is SymFail, a dataset of 536 human-annotated failure trajectories drawn from real multi-agent runs, each one labeled with a graph-linked location, a failure category, and the trace evidence that supports the label. That's a meaningfully different artifact than the pass/fail benchmarks most agent evaluation work relies on. Knowing that a run failed tells you almost nothing about where or why it failed. SymFail is built specifically so that a repair method can be judged against ground truth about what actually broke, not just whether the final output happened to look right.
The uncomfortable numbers
With SymTrace and SymFail in place, the authors ran a large-scale empirical study across three mainstream MAS frameworks, and the headline results are worth sitting with. Existing unguided rerun-based repair methods reproduced the original failure only 67.97% of the time, and successfully repaired the underlying problem in just 6.90% of cases. Read that again: two-thirds reproduction, under seven percent repair. Most of the "successes" that rerun-based debugging claims in practice are not causal fixes — they're the tail of a probability distribution.
The authors then tested a symptom-driven intervention method, one that uses the failure's actual symptom evidence to target where and how it intervenes, rather than blindly rerunning downstream from an arbitrary point. That method repaired 20.15% of the failed cases — a 191.89% relative improvement over the rerun baseline. Roughly triple the repair rate.
Two things stand out. First, tripling a repair rate is a real, meaningful gain — worth building toward. Second, going from 6.9% to 20.15% still means roughly four out of five labeled failures are not being fixed even by the better method. Neither number should be read as "solved." Both are useful precisely because they're honest.
What this means if you're building agent debugging tools
I lead generative and agentic AI work in production, and the instinct to "just rerun it and see" is one I've had to actively resist, not because it's always wrong, but because it's cheap to do and easy to mistake for progress. This paper gives that instinct a number, and the number is under 7%. If your monitoring dashboard shows a failed agent trajectory later succeeding on retry, that outcome alone tells you almost nothing about whether the underlying cause is understood, let alone fixed.
The practical takeaway isn't "stop rerunning things." It's that rerun-based repair needs a causal baseline before anyone can credibly claim credit for a fix, and that symptom evidence — knowing where in the trace things actually went wrong, not just that they did — is what separates a method that triples the repair rate from one drawing lucky samples. For anyone shipping multi-agent systems past the demo stage, that's the difference between a debugging tool and a coin flip with better UX.