Why prompt injection defenses fail, and what token-level feedback fixes

Prompt injection is the failure mode every team running tool-using agents on untrusted data eventually has to reckon with. An agent reads a webpage, a file, or an email, and somewhere in that content is a line like "ignore all prior instructions and do X instead." If the model treats that line as an instruction rather than as data, the attacker now controls the agent. A new paper out of Berkeley, SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation (Yibo Peng, Long Lian, David Wagner, Sizhe Chen — accepted to EMNLP 2026), gets at why our current defenses are so brittle against adaptive attackers, and proposes a fix that is conceptually simple: stop grading the whole response and start grading the tokens.

The problem with sequence-level defenses

Most defensive fine-tuning for prompt injection today runs on DPO or GRPO — preference optimization or policy-gradient methods that compare a "secure" completion against an "insecure" one, or reward a whole rollout based on whether it complied with the injected instruction. That's a sequence-level signal: one scalar judgment for an entire output.

The SecOPD authors point out the obvious problem once you say it out loud. A model's response to an injected prompt is not uniformly good or bad. Early tokens might correctly continue the legitimate task, and the failure only shows up a dozen tokens in, at the exact point where the model pivots to the attacker's instruction. If you reward or penalize the entire sequence as one unit, the optimizer has no way to isolate which tokens actually caused the compromise. It can only nudge the whole distribution in a vague direction, which is a weak, high-variance training signal. The paper reports this shows up empirically as near-100% attack success rates against adaptive injections even after this kind of defensive tuning — including for Meta-SecAlign, the prior state of the art.

"Adaptive" is the important qualifier here. Static injection benchmarks are old news; models can be tuned to resist a fixed catalog of injection phrasings without acquiring any real robustness. Adaptive attacks specifically search for phrasings that evade whatever defense is in place, which is a much closer proxy for what a motivated attacker will do against a deployed system. That's the regime where sequence-level defenses collapse.

Token-level feedback via distillation

SecOPD's fix is to bring in a reference signal that's precise at the token level. Here's the training loop: the policy model gets an injected sample and produces a rollout — its actual output on the compromised input, token by token. Separately, the initialization model (the pre-defense checkpoint) is run on the corresponding clean version of that same input — the one without the injected instruction — and used to score what each of those tokens should look like. The policy's rollout tokens are then distilled against that clean-input reference, token by token, rather than judged as a single pass/fail sequence.

The effect is that the model gets a dense, fine-grained gradient telling it exactly where its behavior diverges from what an uncompromised model would have done, instead of one bit of feedback for an entire response. That's a much more informative training signal, and it's the kind of thing you'd expect to generalize better precisely because it's teaching local behavior (don't let this token drift toward the injected instruction) rather than a global heuristic (this whole response was bad, figure out why).

The results

On a defended Qwen3.6-27B, SecOPD reports a 9.0% attack success rate against PISmith, the paper's state-of-the-art adaptive prompt injection suite, compared to 94.0% for Meta-SecAlign under the same evaluation. That's not an incremental improvement — it's the difference between a defense that mostly fails and one that mostly holds, against attacks specifically designed to be adaptive.

The generalization result is the part I'd flag for anyone building agents rather than reading papers for their own sake. SecOPD was trained on prompt injection data, but the authors also evaluate it on agentic tool-calling — a domain unseen during training — and it gets a 4.7% ASR versus 5.5% for Meta-SecAlign. Worth being precise about what that means: the token-level approach still generalizes to a genuinely different setting, but the margin there is far tighter than the 9% vs. 94% headline. Tool calling is arguably the higher-stakes domain (an injected instruction that gets a tool called with attacker-controlled arguments is a more direct compromise than an injected instruction that gets echoed in text), so it's worth watching whether that gap widens or closes as the benchmark matures, rather than assuming the flagship number transfers one-to-one.

Why this matters if you're shipping agents

If you're deploying an agent that touches external content — retrieved web pages, email, file uploads, tool outputs from other services — prompt injection isn't a hypothetical, it's the threat model. Most teams currently lean on system-prompt hardening, output filtering, and hoping the base model's RLHF holds up. SecOPD is a reminder that the underlying defensive fine-tuning recipe matters as much as the prompt scaffolding around it, and that the granularity of the training signal is not a minor implementation detail — it appears to be the actual bottleneck in why prior defenses degrade so badly under adaptive pressure.

The practical takeaway isn't "go retrain your model with SecOPD tomorrow." It's that if you're evaluating a vendor's or open model's injection robustness, ask what training signal produced it, and demand adaptive-attack numbers, not static-benchmark numbers. A model that resists a fixed list of injection strings tells you very little about what it does against an attacker who's iterating against your specific defense. Sequence-level DPO on a canned adversarial set will look fine on a leaderboard and still fail in production; that gap is exactly what this paper measures.