VeriCUA: when the verifier has to go look for itself

Every team that trains or evaluates computer-using agents (CUAs) eventually runs into the same problem: the agent says it is done, the final screenshot looks right, and the task still failed. The edit was never saved. The file went to the wrong directory. Only half the instruction was carried out. If your verifier accepts that run, you get a false positive, and if that verifier feeds a reward model or a data-curation filter, the mistake gets trained into the agent.

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents, from a Google and University of Hong Kong team, goes after this directly. Its claim is simple: verification should be an agent too. Instead of judging a fixed set of frames in one model call, the verifier should go and look for evidence. I think the framing is right. The results are real but more modest than the framing suggests, and the cost is not small.

Why one-shot judges miss things

The paper names the two standard approaches and the blind spot each one has. A VLM judge, in the style of Pan et al. (2024), sees only the observations it is handed. If the decisive effect isn't in those frames, the judge can't see it. A generated verifier, in the style of EvoCUA, writes a program to inspect application state. But a program written without knowing the application can query the wrong table or the wrong path.

Both fail for the same underlying reason: they commit to a view of the environment before they understand it. If you have built evals for agents in production, you know the pattern. The judge is only as good as the evidence it was given, and nobody checked whether that evidence could answer the question.

The procedure

VeriCUA runs after the executing agent finishes, in four stages.

1. Inspect context. Before writing any rubric, the verifier looks around. Online, that means application packages, directories and database schemas. Offline, it means the recorded steps, frames and accessibility trees.
2. Synthesize a rubric. It turns the instruction into hard and soft requirements. The paper draws a careful line here. Context tells the verifier where to look, but it never adds new conditions for success. A discovered storage directory is somewhere to check, not a requirement.
3. Explore. For each requirement that is still unresolved, the verifier picks a GUI check or a code check across three views of state. The trajectory is what the agent saw and did. The surface is a fresh screenshot or UI dump. Persisted state is files, SQLite stores and settings. The tools are read-only, enforced by an allowlist and a digest of the read targets taken before and after.
4. Arbitrate. Fixed rules, not a model, turn the findings into requirement statuses and then into a verdict: success, fail or abstain.

For me, the most important design choice is the evidence discipline. Every finding has to cite a tool call that actually ran, and the harness rejects citations that don't match one. Each observation is labelled attested (a post-action frame, a file read, a database query) or asserted (the agent's completion message, a pre-action frame showing what the agent intended). Asserted evidence can point the verifier at something to check, but it can never mark a requirement as met. That one rule stops the agent from grading its own homework.

What the numbers say

The abstract is vague about results ("improved agreement," "largest offline gains on hard cases"). The tables in the paper are more concrete.

Online (AndroidWorld, OSWorld, ScienceBoard; gold is each benchmark's official evaluator), VeriCUA has the highest accuracy and lowest false-positive rate in all three environments. Its false-positive rate on AndroidWorld is 7.7% against 16.9% for the auto-generated verifier. On ScienceBoard it is 5.4% against 21.7%, where generated scripts tend to probe the wrong state in unfamiliar scientific software. Across the online environments, it rejects about 80% of the agent's false completion reports.

There is a catch. Some of that precision comes from abstaining. VeriCUA decides only 70–82% of episodes, and abstentions count as fail. On OSWorld its recall (71.3) is below the generated verifier's (78.1). The authors present this as conservative selection for data curation, which is a reasonable choice for a reward signal. It is less reasonable if you need a verdict on every run.

Offline, the gains get thin. Here are VeriCUA's accuracy margins over the best baseline:

• OSReward-Hard: 70.4 vs 65.7 (+4.7)
• OSReward full: 87.2 vs 83.7
• WebArena: 82.7 vs 81.3 (+1.4)

On WebArena, 23% of trajectories end up decided by the single-pass fallback judgment, because the recording simply doesn't hold enough evidence to explore.

The ablation I find most convincing is on online AndroidWorld. Adding code-based state reads to GUI-only inspection raises the decision rate from 45% to 82%, and false positives stay at 3/41. Every additional correct decision is backed by a file or app-store read. That is the hybrid action space doing what it should.

The caveats I would weigh

Cost. On OSReward, VeriCUA spends 345K–378K tokens per trajectory. The three baseline judges together spend 33K–35K. Online it is roughly 39K tokens and 97 seconds per episode, against 16K and 60 seconds for the generated verifier. At RL scale, a 10x verification bill matters.

A stronger single call closes most of the gap. VeriCUA is on Gemini 3 Flash throughout the headline results. In the appendix, a single-call judge on Claude Opus 4.8 scores 69.7 on OSReward-Hard, against VeriCUA-on-Flash at 70.4. On the full OSReward set, several frontier single-call judges (around 89–90) score above VeriCUA's 87.2. The matched-budget comparison in the main text covers only TRM and CUAJudge on the Flash backbone. So the question I'd want answered is how agentic verification compares with a bigger model reading more frames, at equal cost.

Gold labels are imperfect. The authors flag disagreements with reference labels, and a blinded adjudicator sides with VeriCUA on 63% of them. That fits known evaluator errors in OSWorld. It also means some of the measured "accuracy" ceiling belongs to the benchmark, not the verifier.

Where I land

The lasting contribution here is not a benchmark delta. It is the contract: requirements grounded in observed context, reads tagged with where they came from, completion claims treated as claims, and deterministic arbitration you can re-run without new inspection. That is what makes a reward signal auditable. It fits with related work such as evidence-first agentic verification and agentic reward modeling for GUI agents.

If I were building CUA reward pipelines today, I would take the attested/asserted rule and the persisted-state reads right away. I would hold off on full multi-path exploration until someone shows it beats a stronger single judge at matched cost. Verification should cost less than the work it checks, and for now VeriCUA's main argument is that it is more trustworthy, not that it is cheaper.