Agentic abstention: when an agent should stop instead of act
A new benchmark across 28,000+ tasks shows most production agents can abstain but do it too late, that more capable models often stop later, and that a training-free context method called CONVOLVE more than doubles timely stopping on WebShop.
The failure modes that worry me most in production agents are not the dramatic ones. A tool call that errors out is easy: you catch it, retry, log it. The expensive failure is quieter. An agent that should have stopped three turns ago keeps searching, keeps browsing, keeps reformulating the query, accumulating tokens and confidence on its way to a confidently wrong answer. The task was never achievable. The agent never noticed.
A new paper from Han Luo, Bingbing Wen, and Lucy Lu Wang, Agentic Abstention: Do Agents Know When to Stop Instead of Act?, gives this failure a name and, more usefully, a measurement. The short version: most of the agent systems we deploy can technically abstain. The problem is that they abstain at the wrong time, or never.
Abstention is not a single decision
Classic LLM abstention is a one-shot question: given a prompt, does the model answer or say "I don't know"? You can evaluate it with a single forward pass and a calibration curve. Agentic abstention is a different shape of problem. The authors frame it as sequential. At every turn, an agent can answer, abstain, or gather more information. Whether it should abstain often is not knowable up front; it becomes clear only after the agent has interacted with the environment and seen that the world does not contain what the instruction assumed.
That distinction matters because it moves the decision from "do I know the answer" to "is further interaction still worth it." Those are not the same question. An agent can be perfectly calibrated about its own knowledge and still burn twenty tool calls discovering that no product matching the user's constraints exists in the catalog.
The measurement, and the brutal middle
The study spans web shopping, terminal environments, and question answering: 13 LLM-as-agent systems and 2 scaffolds, evaluated across more than 28,000 tasks. The headline is not whether agents can abstain but when. Some never abstain when they should. Others get there eventually, but only after a long tail of unnecessary interactions.
The hardest cases are the ones that look feasible until the environment says otherwise: a shopping instruction that seems perfectly ordinary until you discover that no listing satisfies it. Up front there is no signal to abstain. The signal arrives mid-trajectory, and the agent has to notice it against its own momentum to complete the task.
I would flag this as the single most operationally relevant finding. In real deployments, the plausible-but-unachievable task is the common case, not the edge case. Users ask for things that almost exist.
Bigger models, later stops
The result that should give us pause: model scale, reasoning, and scaffolding all affect abstention, but not in the direction you would hope. Larger or more capable models sometimes do worse at timely abstention.
This is less surprising than it first sounds. A more capable model is better at finding a path forward: better at reformulating, at trying one more tool, at constructing a plausible answer from thin evidence. That same competence is exactly what suppresses the instinct to stop. Capability and calibration about when to stop are different axes, and we have been optimizing hard for the first while assuming the second comes along for free. It does not.
CONVOLVE: stopping rules as context, not weights
The part I find most practical is the proposed fix. Rather than fine-tune for abstention, which is expensive and which the scaling result suggests may not even point the right way, the authors introduce CONVOLVE, a context-engineering method. It takes full interaction trajectories and distills them into reusable stopping rules, then puts those rules back into the agent's context.
No parameter updates. On WebShop, it raises Llama-3.3-70B's timely recall rate from 26.7 to 57.4, better than doubling, purely by giving the model distilled experience about when continued interaction stops paying off.
This lands for me because it matches what already works in production. The leverage in agent reliability is increasingly in the context layer, what you retrieve, what you remind the model of, what you put in front of it at decision time, not in the weights. A stopping rule distilled from a thousand prior trajectories, of the form "if the top results still do not match the constraint after N refinements, abstain," is exactly the kind of operational knowledge that is hard for a model to derive from scratch each run but easy to hand it.
What I am taking from this
Three things change in how I would build and evaluate agents after reading this.
First, abstention timing belongs in the eval harness. Most agent benchmarks score task success and maybe cost. Neither captures the agent that eventually gives up after fifteen wasted turns. Timely recall, did it stop at the right moment, is a separate axis and needs its own number.
Second, more capable does not mean more trustworthy on this axis. If you upgrade the base model, re-measure stopping behavior. Do not assume it improved.
Third, treat the stop condition as something you engineer, not something you hope emerges. CONVOLVE is one way; explicit budgets, environment-derived feasibility checks, and distilled stopping rules in context are all cheaper than a fine-tune and more controllable than trusting the model's instincts.
The instinct to act is the easy part to instill in an agent. Knowing when further action is futile, and stopping cleanly, turns out to be the harder and more valuable skill. This is the first work I have seen that measures it at scale and hands us a lever to improve it without touching the weights.