Ai2 ships BenchMIRT, a per-question audit of what benchmarks actually measure

Ai2 published BenchMIRT this week, alongside a technical report and open-source code — a method for auditing LLM benchmarks at the level of individual questions, not just the aggregate score everyone quotes in a model card. I've spent enough time staring at eval dashboards to recognize the itch this scratches: a benchmark's name promises one thing ("safety," "reasoning") but the number it produces is quietly a blend of several things, and until now there was no clean way to tell how much of each.

What a single score hides

Take BBQ, a benchmark built to test whether a model leans on social stereotypes. One item asks about a grandson and grandfather trying to book an Uber together. It's probing age bias, sure — but answering it correctly also requires tracking who's who in the scenario and reasoning from the evidence given rather than a stereotype-driven guess. Two abilities, one score. WildJailbreak has the same problem in a different shape: it mixes harmful jailbreak prompts with benign prompts designed to catch over-refusal, and averaging the two into a single number erases the distinction between "resists misuse" and "doesn't refuse legitimate requests."

Ai2's answer draws on item response theory (IRT), a psychometrics technique for measuring latent ability from patterns of test responses — the premise that not every question carries the same signal, and some discriminate between strong and weak performers far better than others. Ai2 had already applied single-dimensional IRT to individual benchmarks in its Fluid Benchmarking work. BenchMIRT is the multidimensional extension: instead of assuming one latent ability drives a benchmark, it lets several candidate capabilities compete to explain the same pattern of responses.

Two dimensions fell out, unprompted

The team trained BenchMIRT on results from 100 LLMs across 16 benchmarks and roughly 34,000 questions — six general-reasoning benchmarks (MMLU-Pro, GPQA, MATH, BBH, and others) and ten drawn from Ai2's Olmo 3 safety suite (HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, XSTest). Critically, the model was never told which benchmark belonged to which category. It independently recovered two dominant latent dimensions — safety and general reasoning — and reproduced the same split when the analysis was rerun from scratch.

Applied back to the individual benchmarks, that decomposition is where it gets useful. BBQ, despite being marketed as a safety benchmark, correlated far more strongly with the reasoning dimension than the safety one — a low BBQ score may say more about a model's ability to parse an ambiguous scenario than about its social bias. WMDP, which tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, also tracked reasoning more than safety, and in the counterintuitive direction: stronger reasoners scored lower, because the benchmark counts refusing to answer as the correct response. HarmBench split down the middle — its standard and contextual harm-compliance questions (write a phishing email, act on this malicious instruction) loaded onto safety, while its copyright-violation questions (reproduce these song lyrics) loaded onto reasoning.

None of this means these benchmarks are broken. It means a single reported score is a mixture, and BenchMIRT gives you the mixing weights.

A useful side effect: fewer questions, same signal

Because BenchMIRT scores each question's difficulty and how well it discriminates strong models from weak ones, Ai2 used it to prune the question sets: keeping the top 10% of the most informative questions per benchmark preserved nearly the same ranking of models on the underlying capability as the full set, and 50% matched it even more closely. Held-out prediction of whether a model would answer a specific unseen question correctly hit 79% accuracy, against 70% for a baseline that just assumes a model performs at its overall benchmark average.

My take

The most useful part of this work isn't the compression — it's the admission that "benchmark score" is a proxy variable, not a direct measurement, and that the proxy's composition changes from benchmark to benchmark. Every team running evals against a production model already suspects scores drift for reasons unrelated to the capability they're supposed to track; BenchMIRT puts a number on that suspicion. The tradeoff Ai2 names honestly is the one that matters operationally: the same question-level transparency that lets you build a leaner, more interpretable benchmark also tells someone exactly which questions to strip out to make an unsafe model look safe on paper. That's not a reason to skip the audit — it's a reason to start treating benchmark composition as a security-relevant artifact, not just a scoring convenience.