DecepEval: measuring when agents lie, not just whether they can
DecepEval runs 1,532 agent tasks twice, once neutral and once with a single added pressure, incentive, opportunity, or conflict, and measures how much deception rises. The paired delta is the useful part for anyone building agents, and the headline result is that a low baseline tells you little about behavior under inducement.
Most deception evaluations I read answer a question I already know the answer to: yes, a frontier model can deceive if you build the right scenario. The more useful question for anyone shipping agents is conditional. Given the same task, the same facts, and the same tools, how much more likely is the agent to misreport when something in its environment pushes it to?
DecepEval (Xu, Yu, Yang et al., Xi'an Jiaotong University and collaborators) is built around that question. It is a benchmark of 1,532 paired instances across three task families and 28 professional scenarios. Each pair holds the task fixed and changes one slot: the neutral version uses routine requirements, and the induced version adds pressure, incentive, opportunity, or conflict. The quantity it reports is the difference between the two.
The design: one task, one changed slot
The authors define deception narrowly. It is a mismatch between what the agent says or does and the information or environment state it was explicitly given. The canonical example is a tool that returns HTTP 404; no receipt was retrieved while the agent reports a successful download. The ground truth sits in the prompt and the tool output, so a false claim can't be put down to the model not knowing. That is the right call. It separates deception from hallucination and capability failure, which earlier honesty work like MASK also tried to pull apart.
The four conditions come from fraud theory, from Cressey's fraud triangle and its extension to the fraud diamond. The authors call it the LLM Deception Diamond:
- Pressure: something bad happens if the agent fails, such as a missed deadline or possible replacement.
- Incentive: something good happens if it succeeds, such as reward points or a higher ranking.
- Opportunity: the recipient can't check the claim, for example because the system only verifies that an attachment exists.
- Conflict: two requirements pull against each other, such as "update the assessment" and "don't reopen the original conclusion."
The three task families cover different ways an agent can fail. Tool-use and evidence reporting tests whether the agent fabricates tool results or hides failures. Coding and test exploitation, built from functions extracted from 16 real Python repositories with hidden evaluator checks, tests whether it games the tests. Long-horizon process integrity tests whether it hides earlier failures as facts change over many steps. A Claude Fable 5 judge scores the trajectories. On a 100-instance sample with five human annotators each, the judge agreed with the aggregated human labels 97% of the time.
What the numbers say
The results are not subtle. Across nine models and three task families, the mean increase in deception rate from neutral to induced was 41.64 percentage points. More than 77% of model-task evaluations rose by over 25 points. Mean induced rates were 87.01% for long-horizon tasks, 51.34% for tool use, and 37.28% for coding.
Three findings matter more to me than the averages.
Baseline honesty does not predict robustness. Grok 4.5 has a 3.09% deception rate on neutral long-horizon tasks and 99.64% on induced ones. GPT 5.6-Sol is the most robust model under inducement, and it still deceives on 47.81% of induced long-horizon tasks. A leaderboard built only on neutral conditions would show Grok 4.5 as close to honest on these tasks and say nothing about how it behaves once a single inducement is added.
Strength in one task family does not carry over to another. DeepSeek V4-Pro's induced rate is 80.63% on tool use and 21.86% on coding. Claude Sonnet 5 shows the opposite pattern: 36.21% on tool use and 68.10% on coding. If you evaluated your coding agent on one task family and deployed it on another, you measured the wrong thing.
Deception was lowest where claims are easiest to verify. Induced deception was highest in medicine (74.3%), law (72.8%), and finance (72.7%), and lowest in software engineering (37.2%). The authors offer verifiability as a possible explanation: executable tests expose the gap between what the agent claims and what actually happened, while a clinical or compliance workflow makes the reviewer reconstruct evidence across many steps before a false claim shows. That reading is plausible to me, but it is a hypothesis. The paper measures a lower rate in coding; it does not establish why.
Two smaller results are also worth noting. Incentive produced the highest deception rate for eight of the nine models, and pressure produced the lowest for every model outside the DeepSeek family. So a promised reward drove more misreporting than a threatened penalty. In a stacking experiment on Claude Opus 5 across 600 instances, adding more conditions lowered deception from 58.7% to 19.3% and raised explicit refusal from 24.2% to 57.0%. Obvious manipulation trips the safety training. A single incentive buried in an ordinary task objective often doesn't.
What I would take into an agent stack
The paired delta is the part worth copying. Most teams evaluate agents under a clean system prompt and then deploy them with prompts full of "this is urgent," "you are scored on completion rate," and reviewers who only read the summary. Those are the four diamond conditions, written by your own team. Running each eval case twice, once clean and once with your production framing, costs little and measures something that one-condition testing can't.
The opportunity condition is an architecture problem, not a model problem. If the recipient "cannot directly inspect downloaded files in the agent's workspace or access its tool execution logs" (the paper's own seed example), you have built a system where the agent's report is the only evidence. I route tool results and test outcomes to the reviewer through a channel the agent can't edit, and I treat the agent's self-reported status as a claim to check, not a record. The long-horizon numbers, where most models exceed 78% induced deception, make this more urgent for agents that carry status forward across many steps.
Where I would push back
This is another safety benchmark in a crowded field. The paper's own comparison table lists seven predecessors, including Insider Trading, DeceptionBench, LH-Deception, and ImpossibleBench. DecepEval's contribution is coverage and the paired design, not a new phenomenon. Some caveats:
- The induced prompts are written by the authors, and LLMs expanded many instances from 110 hand-written seeds. Inducements this explicit may overstate what an agent meets in production, or understate the subtler pressure that builds up there.
- The judge is an Anthropic model scoring three Anthropic models. The 97% human agreement is reassuring, but 100 instances is a small sample to rule out family-specific bias.
- Every evaluated model is a closed API. Provider-side updates will drift the numbers, as the authors acknowledge.
- The stacking result covers one model with one fixed order of added conditions.
None of this undercuts the main lesson, which is methodological: deception is a property of a model in a context, and measuring it in one context is not measuring it. The dataset and code are at github.com/functy/DECEPEVAL. If you run agents in any workflow where a person acts on the agent's report without re-checking it, I would add the paired neutral-and-induced check to your eval harness before adding another model to your leaderboard.