Which agent should you improve first? IF-MAP measures leverage, not blame

Every multi-agent system I have shipped eventually reaches the same meeting. Accuracy is flat, the budget is finite, and someone asks which agent we should work on next. The usual answer comes from failure analysis: read the traces, find the agent that most often holds the bag when things go wrong, fix that one. A new paper, IF-MAP: Influence-Function-Inspired Agent Prioritization in Multi-Agent Systems (Mao et al., CUHK-Shenzhen, Harbin Institute of Technology and Harvard), argues that this answers a different question than the one we are asking, and offers a cheap, black-box way to answer the right one.

Blame is not leverage

The authors frame the problem as agent prioritization: given a fixed workflow, pick the agent whose improvement will produce the largest end-to-end gain, without running the full improvement procedure on every candidate. The key observation is that responsibility for failures does not determine improvement potential. An agent can show up in many failure traces and still have little room to get better, while a quiet agent upstream may admit a change that lifts everything downstream. Their related-work section points to CICA, which found that the module assigned the most responsibility is not necessarily the one whose repair helps most.

This matches my experience. Failure attribution, such as the Who&When line of work, is a debugging tool. It tells you where things broke. What a roadmap needs is a sensitivity measure: where does a unit of effort move the metric?

The method: a finite-difference gradient for prompts

IF-MAP borrows its intuition from influence functions, which estimate how perturbing training data changes a model's loss. Here, the perturbation is applied to an agent's prompt, and the response is the whole system's loss. No weights, gradients or Hessians are needed.

The procedure for each agent:

1. Perturb. Build up to ten formal semantic edits to that agent's prompt. These are clarifying the downstream interface, resolving an ambiguity, adding a verification step, handling edge cases, and so on. The role and output format stay intact, and every other agent, model and edge stays frozen. The ten are not written in one pass. They come out of an adaptive construction step that generates up to 36 auxiliary candidates (12 initial mechanisms, three slots each), and each candidate is screened with one development evaluation on a shared subset of about a quarter of the probing set. Reproducing the results means running that screening as well as the ten formal probes.
2. Run end to end. Execute the full workflow on a held-out probing set and record the change in mean loss. Because the whole pipeline runs, the measurement includes effects that propagate through downstream agents.
3. Embed the change. Represent each edit as a displacement in prompt-embedding space (Qwen3-Embedding-8B, 512 dimensions). Normalize both the direction and the loss change by the embedding distance, so a large rewrite does not dominate just because it is large.
4. Fit a local response. With about ten probes in 512 dimensions, an unconstrained fit is hopeless. They project onto a low-dimensional subspace spanned by the observed directions and fit ridge regression (λ = 1.0) to get a local slope vector g.
5. Score. The priority is the norm of g. By Cauchy–Schwarz, the fitted model predicts a maximum first-order loss reduction of r‖g‖ for a step of radius r in the probed subspace, so ‖g‖ is that prediction at unit radius. It is a model prediction, not a guaranteed improvement. In plain terms, it estimates how steep the loss surface is around this agent.

As a physicist I find this appealing. It is linear response theory with a finite-difference estimator, applied to a system whose state space is natural language. The embedding gives the perturbations a geometry, and the ridge fit turns a handful of noisy probes into a usable estimate.

What it buys

The evaluation uses six AFlow-derived workflows: HumanEval, MBPP, NQ, HotpotQA, DROP and MATH. Qwen3-8B executes the workflows. The same priorities are then tested under two kinds of improvement.

• Prompt optimization of the selected agent gives a macro-average gain of 3.81 pp, against 1.85 pp for picking an agent uniformly at random and 3.30 pp for FAMAS, the strongest of six adapted failure-attribution baselines. It beats the uniform reference on all six workflows.
• Model upgrades, swapping the selected agent's model for one of five stronger targets including Qwen3-32B, DeepSeek-V3.2 and GPT-5 nano, give macro-average gains of 7.62–16.42 pp, against 3.81–8.62 pp for uniform selection. That is 1.86–2.05× the uniform gain. The priorities were computed once from prompt probes and never re-tuned per target model.
• Ranking quality. Mean Kendall τ_b between the priority ranking and the realized-gain ranking is 0.741 for prompt optimization and 0.708–0.804 across upgrades.

The transfer result is the one I would underline. Leverage found by perturbing prompts predicted where a model swap would pay off. That suggests the probes measure something structural about the agent's position in the graph, not just how well its prompt happens to be written.

Cost: where the tokens go

Probing is not free. Each perturbation runs the whole workflow, so IF-MAP uses more base-model executions than attribution methods that read a trace once. But it makes far fewer calls to the expensive diagnostic model, and the real savings show up downstream. Compared with optimizing every agent, picking one agent with IF-MAP uses 25.8% of the Qwen tokens and 64.8% of the DeepSeek tokens. Compared with upgrading every agent, it uses about 35–36% of the target-model tokens. The pattern is sound: spend cheap compute to decide where the expensive compute goes.

Caveats worth taking seriously

• Small graphs. The workflows have two to five intervenable nodes. With two nodes (DROP), almost every method ties. The real test is a 15-agent production system with tool calls and loops, and the paper does not run one.
• Probe budget matters a lot. With three probes per agent the prompt-optimization gain falls to 2.45 pp and τ_b to 0.29. Five probes are still clearly worse than ten. The paper tests only three budgets, so this does not establish a threshold, but it does say the small-budget variants should not be trusted.
• Top-1 versus full ranking. A simpler score (mean absolute probe effect) had slightly higher ranking agreement but lower top-pick gain. If you only act on the top agent, judge the method on top-1 gain.
• Local and linear by construction. The score depends on the embedding model, local linearity and probe coverage, which the authors acknowledge. A large intervention such as a full model swap is not a small step, so the transfer result is empirical rather than guaranteed.

How I would use it

I would treat IF-MAP as a prioritization pass that runs before the improvement loop, not as a replacement for trace-level debugging. Attribution tells you what broke. Sensitivity tells you what will move. In a production workflow I would run about ten structured perturbations per agent against a frozen eval slice, rank by response norm, and spend the quarter's optimization or model budget on the top one or two agents. Then I would re-probe, because once you improve one agent the loss surface around the others changes.

The code is on GitHub. The full paper includes the ten-slot perturbation spec and complete cost tables, and both are worth reading before you build your own.