How much should you trust an LLM to grade your agent?

If you run agents in production, you already know the quiet tax: evaluating what the agent actually did. A single trajectory on a serious agentic benchmark can take a human annotator hours to grade step by step. That cost is why almost everyone has quietly handed the job to another model — LLM-as-a-judge (LLMJ), critiquing agent trajectories at the process and outcome level, at scale. The part we tend not to say out loud is that the judge's verdicts usually go unchecked. We measure the agent against the judge, and we measure the judge against nothing.

A new paper, Counsel: A Meta-Evaluation Dataset for Agentic Tasks (Pisupati et al., June 2026), takes that gap seriously and does the unglamorous thing: it measures the judge. It is the first public meta-evaluation dataset for agentic tasks, and the headline number is one worth sitting with — even the strongest open-weight judge they tested agrees with humans only about 65% of the time on reasoning, against roughly 88% on where the error is.

What Counsel actually is

Counsel is built in two layers. The first layer is process-level critiques produced by open-weight LLM judges on two existing agent benchmarks: tau-bench (customer-support agents) and DA-Code (coding agents). The second layer is the part that makes it a meta-evaluation: human annotators went through those critiques and graded them.

The grading scheme is the clever bit, because it refuses to collapse correctness into a single yes/no. For each error the judge flagged, a human labels it as one of three things: spot on, correct location but poor reasoning, or should not have flagged. That separation — did you point at the right step, and did you explain it correctly — is what lets the dataset stratify a judge's critiques along two independent axes. The annotators reached a Krippendorff's alpha of 0.78, which is solid inter-annotator agreement for a task this subjective, so the labels are not noise.

The number that should make you pause

The authors compare open-weight judges and find, sensibly, that more capable models and more reasoning effort both push agreement with humans up. But the ceiling tells the real story. The strongest judge reaches about 88% agreement on error location and about 65% on reasoning quality.

Read that as two different claims, because they are. The judge can usually point at the step where the agent went wrong. But more than a third of the time, the story it tells about why does not survive human scrutiny — even when it pointed at the right step. The judge is good at "something broke here" and noticeably worse at "here is what broke and why."

Why location and reasoning come apart

This split is not surprising once you name it. Flagging the location of an error is close to a pattern-recognition problem: a tool call returned an error, a required field is missing, the final answer contradicts an earlier step. Those are local, often surface-level signals. Explaining the error correctly is a causal-reasoning problem over a long, branching trajectory — you have to reconstruct what the agent intended, what state it was actually in, and which of several plausible causes is the real one. The first is detection; the second is diagnosis. We have known for a while that language models are stronger at the former than the latter, and Counsel quantifies exactly how wide that gap is in the agent-evaluation setting.

That more reasoning effort helps but still tops out around 65% is the other thing to internalize. There is real signal in a judge's rationale, but it is not the kind of signal you can treat as ground truth without a calibration step.

What this changes about how you use a judge

I lead generative and agentic systems in production, and this lands on a practical decision a lot of teams are making right now without measuring it. Two uses of LLMJ are especially exposed.

The first is scoring — using a judge to rank agent variants or gate a release. If your metric leans on the judge's location calls, you are on reasonably firm ground; ~88% is usable as a coarse signal. If your scoring weights the judge's stated reasoning — penalizing a trajectory for the specific failure the judge described — you are building on a number near 65%, and you should treat those scores as hypotheses, not facts.

The second, and the one I would worry about more, is curating training data. A growing pattern is to mine agent trajectories with an LLM judge and feed the critiques back as supervision — reward signals, preference pairs, or rationales for distillation. If you train on the judge's reasoning as if it were correct, you are baking a one-in-three error rate directly into your labels, and the model will happily learn the judge's confident-but-wrong explanations. Filtering on location is safer than trusting the rationale.

The honest move is to do what Counsel makes possible: stratify your judge's critiques, calibrate against a human-labeled slice, and know your judge's location-versus-reasoning profile before you wire it into a loop.

Why I think this matters

Counsel is generated entirely with open-weight models and released under a permissive license, which means anyone can reproduce it, extend it to new benchmarks, or use it to fine-tune better judges. That reproducibility is not a footnote; it is most of the value. The field has spent two years building agents faster than it can trust the tools that grade them, and "the soundness of LLMJ critiques often goes unmeasured" is a fair description of the gap.

The contribution here is not a better judge. It is the discipline of measuring the evaluator instead of assuming it — and a public dataset that lets the rest of us do the same. If you are using an LLM to grade your agents, the useful takeaway is small and concrete: trust where it points more than why it points there, and verify before you trust either.