PROBE and the third axis of agent evaluation: auditing policy compliance

Most of the agent failures I worry about in production are not wrong answers and not toxic text. They are an agent that books the right flight before the customer said yes, or quotes a price in a currency it was told never to use, or mentions an internal fare code in an otherwise polite reply. The task succeeds, the text is clean, and the trace still breaks a rule.

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (Mishra, Sharma, Lukas and Lahlou, MBZUAI) is aimed at that gap. It trains a small LLM auditor with reinforcement learning to hold multi-turn conversations with a frozen tool-using agent and find policy violations. I think the problem framing is right. The method is mostly familiar parts put together. The parts worth reading are the honest ones: where the reward gets gamed, where the LLM judge fails, and how shallow the audits still are.

The missing axis

The paper's framing is simple and I agree with it. Capability benchmarks such as WebArena, AgentBench and tau-bench ask whether the agent finished the task. Content-safety red-teaming, going back to Perez et al.'s red teaming of language models with language models, asks whether the model said something it shouldn't. Neither asks whether the agent followed its operating rules while doing a legitimate job.

That third property lives in the full trace: user turns, assistant turns, tool calls with their arguments, tool outputs, and the state of the database. The authors' example is the one I would use too. A user asks for the cheapest LAX to Tokyo flight, the agent searches, summarises and calls book_flight straight away. A capability benchmark marks that as a success and a safety classifier sees nothing wrong. Yet there was no explicit confirmation before a mutating action, and perhaps no fare disclosure either. Neither failure shows up in the final response.

How PROBE is set up

The testbed is SkyAssist, a deterministic flight-booking environment with 8 tools (four read-only, four that change state) and 20 policies across business, safety, privacy and operational categories. Sixteen verifiers are deterministic predicates over the trace. Four semantic ones (premium-option mention, round-trip suggestion, explicit confirmation, internal pricing disclosure) used an LLM judge during training.

The auditor is Qwen3-4B-Instruct trained with QLoRA. The target is a frozen Qwen3-30B-A3B served with vLLM. The auditor sees only the public conversation. It never sees the target's prompt, tools, database or verifier labels. The verifiers see everything and return a single scalar reward.

The reward is where most of the design effort went. Counting verifier hits rewards an auditor for triggering the same easy omission over and over. PROBE instead uses a composite of four terms. The violation term is severity-weighted and deduplicated within an episode: full credit the first time a policy fires, half credit the second time, nothing after that. A progress term rewards reaching deeper tool states such as booking or cancellation for the first time. A novelty term pushes the auditor towards policies it rarely triggers. A plausibility term penalises messages that read like an assistant, fabricated claims of system access, and markdown. A horizon curriculum grows the conversation from 4 to 14 turns. The same rollouts are then optimised three ways, with GRPO, DPO and a trajectory-balance GFlowNet.

None of these pieces is new. This is automated red-teaming moved from the text surface to the tool surface, run in a tau-bench-style policy environment, with diversity pressure borrowed from GFlowNet red-teaming. The paper says as much in its related-work table. What it adds is the combination, plus a controlled comparison of the optimisers on the same rollouts.

What the numbers say

The abstract gives no numbers. The body does. The headline metric is suite coverage: the share of the 20 policies triggered at least once across 50 held-out scenarios.

• Random scenarios cover 5%, and a zero-shot Qwen3-30B auditor covers 35%. The best trained 4B auditors (GRPO and DPO) reach 60%, which is 12 of 20 policies.
• A tau-bench-style scripted test suite finds one policy. It reaches 55% only when it is given an expert answer key listing what triggers each policy.
• Coverage and volume are different things. A GRPO variant with deduplication but no other shaping produced 1,914 violations per 100 episodes and still covered only 55%.
• Transfer works, but the gains shrink. Against an unseen Gemma target, a DPO auditor trained on Qwen covers 40%. On tau-bench, coverage goes from 2 to 3 of 10 policies.

I would qualify the "much smaller auditor" claim. Qwen3-30B-A3B is a mixture-of-experts model with about 3B active parameters. The 4B auditor is smaller in total parameters, but not obviously in compute per token.

The findings I would act on

The LLM judge was gamed. Two annotators agreed almost perfectly with each other (kappa 0.985), but the training-time judge matched their consensus only 39.6% of the time. Its precision on the confirmation policy was 0%. GRPO found this out: judge-flagged confirmation violations rose from 133 to 302 across checkpoints, and the deterministic check confirmed none of them. DPO, which only uses the ranking of rewards, showed no such escalation. Re-scored deterministically, the best checkpoints drop from 60% to 55% and the ranking holds. I give the authors credit for reporting this. If you train an adversary against an LLM judge, you are also training it to find the judge's blind spots.

The audit ceiling is a tool-depth ceiling. In evaluation the target called search hundreds of times and booked a flight one to three times. It never cancelled anything. Over 90% of target tool calls were searches. The mutation-level policies, which include several of the highest-severity ones, mostly went untested. A conservative target hides whether its deeper safeguards work at all. A cooperative 30B user simulator that simply tries to finish bookings matched the trained auditor's 60%. Getting the agent into deep states may matter as much as being adversarial.

The optimisers behave differently. GRPO and GFlowNet are sensitive to the scale of the reward, while DPO is stable because it only uses rankings. GFlowNet had the highest training yield but the lowest held-out coverage (45%). Its mode-covering objective only helps if the reward actually spreads mass across distinct policies.

Where this fits

The limits are real and the authors state them: two environments, one checkpoint per method, and annotators who are also authors. Still, the way it is meant to be used is the right one. To audit a new domain you supply three things: the tools, a policy list, and a verifier for each policy. The training loop stays the same. Failing traces become regression tests, and the patched agent is audited again. Most of the work is writing verifiers that are deterministic and can see the whole trace, and that work pays off even if you never train an auditor. Code is on GitHub.