Trial by trace: grading the harness, not just the agent

Most agent benchmarks ask one question: did the task get done? A new paper, Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution, asks a second one that I think matters more for anyone running agents in production: did the harness do its job while the task was getting done?

The headline number makes the case. Across 3,420 runs, 31.1% achieved task success while failing at least one harness criterion. Put differently, in nearly a third of all runs the task passed while at least one harness criterion was judged as failed. The paper is careful to say that a failed criterion is not the same as a confirmed defect. Still, the criteria describe exactly the problems a passing outcome can hide: lost context, a tool result that misreports an exit code, a lifecycle step that never completes.

Agent = model + harness

The paper starts from a simple decomposition: an agent is a model plus a harness. The model picks actions. The harness builds the context, exposes tool schemas, carries calls and results back and forth, keeps session state, compacts history, manages subagents, and enforces permissions. The same model behaves differently under different harnesses, and anyone who has moved a workload from one coding agent to another has seen this firsthand.

Outcome-only benchmarks blend the two contributions. A correct final file can sit alongside a truncated result channel the model never saw. A failed task can come from a model mistake made on perfectly faithful information. If you only score the deliverable, those cases look the same as everything else.

TraceTrial-Bench treats the native harness as the system under test. It organizes harness reliability into seven components: context, execution, governance, lifecycle, observability, tools, and verification. It builds 95 mechanism-oriented tasks around them, with executable or state-based ground truth. The coverage is uneven, and the paper is upfront about it: 27 execution tasks and 19 lifecycle tasks, but only 7 for observability and a single task for verification.

The tasks are designed to put pressure on specific obligations. Examples include oversized outputs that need recovery, nonzero exit codes paired with artifacts that look like success, pending background jobs, facts that have to survive a session boundary, and private values that are only reachable through an interaction channel. These are familiar failure modes for production agent systems, and it is good to see them named as test conditions rather than anecdotes.

The pipeline

The measurement design is the most useful part of the paper. Four pieces stand out.

Native execution, normalized traces. Codex, Claude Code, OpenClaw, and DeepSeek Harness each run through adapters that keep their native prompt construction, tool transport, context policy, and permissions. Each harness's own event stream is mapped into a shared Trace IR, and the raw record is preserved next to it. Tool results are paired with their calls using native call IDs. Ambiguous records stay explicitly unlinked instead of being paired by heuristic.

Evidence authority by boundary. This is the idea I would steal first. A tool result has several representations that are related but not interchangeable: what the process actually did, what the harness passed to the model, what the session archive recorded, and what the evaluator sees. The benchmark assigns authority per question. Environment state decides what happened. The model-facing record decides what the model could observe. Provider-request captures decide which tool declarations were actually sent. A harness-fidelity criterion only becomes possible once you keep the external truth and the model-facing view as separate observations. One of the paper's examples is a process that timed out while the model was never told.

An admission gate before scoring. Container lifecycle, archive integrity, trace scoreability, and oracle availability are checked before any semantic judging. When the measurement apparatus fails, the run is marked inconclusive rather than counted as a harness defect. Many internal eval setups skip this step, and their dashboards absorb infrastructure noise as a result.

Two layers, then attribution. The Agent layer checks deliverables against authoritative state. The Harness layer checks whether the native system met the obligations the task exercised. Failure attribution then connects failed harness criteria to failed agent criteria through mechanism chains declared before execution. The paper is careful here. An activated link means the predeclared endpoints both failed and the activation condition matched. It does not mean propagation was proven. A validated propagation claim needs manual review of event order, an observable intermediate distortion, and competing explanations. The stored label harness_caused means a candidate link fired, and the authors say so explicitly.

What the numbers say

Across 4 harnesses, 9 models, and 95 tasks, macro-averaged harness scores range from 59.8% to 72.0%. The authors also report that task success and harness-criterion satisfaction rank the harnesses differently. That divergence is the real result. It means a team picking a harness on task success alone may be optimizing the wrong axis for reliability.

The 31.1% figure is an operational signal, not a defect count. If even part of it holds up under review, it describes a familiar pattern: a harness shortfall gets absorbed by a capable model this time and surfaces as an incident the next time. A harness that drops a timeout signal can still produce a correct answer when the model happens to retry. That is luck, not reliability.

Where I would be careful

The framing is strong. The evidence needs to be read for what it is.

The harness layer is LLM-judged. Deterministic checks are fixed first, but semantic criteria are scored by a single judge, deepseek-v4-flash, across all 3,420 evaluations. The paper states that it does not estimate judge agreement, selector recall, or candidate-link precision at corpus level. A harness score of 59.8% versus 72.0% is therefore a difference in judged criterion satisfaction, not in confirmed defects. The authors say this themselves, and readers should hold them to it.

Judge budgets are not uniform. Per-criterion output limits were 4,096 tokens in 1,966 evaluations, 8,192 in 1,452, and 32,000 in two. Judge identity is fixed, but the scoring conditions are not identical across cells.

Empty evidence scores as zero. If a selector returns nothing, the criterion fails. That is a defensible convention, but it means some harness zeros reflect what the evidence contract could see, not what the harness did.

Not-exercised mechanisms inherit the agent outcome. A harness rubric marked not exercised resolves to pass only if every agent criterion passed. That couples the two layers, which the paper acknowledges. It also means the layers are not statistically independent, and the 31.1% should be read with that in mind.

One execution per cell. Each of the 3,420 cells holds one selected run, including some replacement executions from earlier recovery. The authors make no claims about within-cell repeatability, and neither should readers. Agent runs are noisy, and single-sample differences between harnesses on a given task may not hold up when rerun.

What I take from it

For agent builders, the transferable lesson is about method, not the leaderboard. Keep the trace and the outcome as separate observations. Record what the environment actually did next to what the model was shown. Gate the measurement apparatus before you score the system. Do not let correlated failures turn into causal claims without someone reading the trace.

The runs that deserve the closest look are the ones where the task passed and the trace still shows a problem. TraceTrial-Bench puts a first, judge-dependent number on how often that may happen. The next step it needs is calibration: human agreement rates for the judge and repeated runs per cell, so the harness scores carry the same weight as the framing behind them.