Inside GeneBench-Pro: a benchmark that measures research taste, not recall

What actually shipped

OpenAI published GeneBench-Pro, a 129-question benchmark built to test something most AI evaluations don't touch: research taste in computational biology and genomics. Not "can the model recall a gene name" or "can it summarize a paper." Can it look at a pile of real-shaped multi-omic data, decide which confounds actually threaten the analysis, pick a defensible statistical approach, and commit to one number it can justify.

Each question comes with synthetic but realistically noisy datasets and ground truth derived from real experimental structure, scored the same way across every lab's models. That cross-lab comparability is the part worth paying attention to. Most benchmarks a lab publishes are self-graded on their own terms. GeneBench-Pro is scored consistently across model families, which is what makes the leaderboard mean something.

The result: GPT-5.6 Sol leads at 31.5%. The next-best model, Opus 4.8, sits at 16%. On a benchmark built around real analytical judgment rather than trivia, the frontier model gets fewer than one in three questions right, and the gap to second place is nearly a factor of two.

What the questions actually look like

The published case studies are worth reading directly, because the framing of "AI benchmark" undersells how specific these problems are. A sample of what a model is handed:

• Somatic oncology. Given a molecular tumor board registry, estimate whether a structural-variant-driven drug target actually confers positive net clinical benefit once you net out an 8-week toxicity discontinuation rate against the treatment effect — and the target subgroup itself has to be reconstructed from long-read sequencing, expression, and pharmacogenomic evidence before the benefit-risk question is even well posed.
• Functional genomics. Decide whether a candidate lncRNA dependency is transcript-specific, or whether the apparent effect is actually driven by disrupting the local DNA locus or repressing a neighboring gene — a question that only resolves once you've controlled for guide GC content, plate effects, and neighbor-gene knockdown.
• Statistical genetics. Two correlated proteins share a genetic locus; estimate each one's direct causal effect on disease risk using cis multivariable Mendelian randomization, correcting for linkage disequilibrium, assay scale, and winner's curse.
• Carrier screening. From assay data alone, back out ancestry-specific carrier frequencies, residual risk after a negative screen, and affected-conceptus risk — where the residual risk explicitly depends on catching pseudogene interference and CNV artifacts that a naive genotype call would miss.

None of these have a single lookup answer. Each one is a chain of judgment calls, and getting the final number right requires getting most of the intermediate calls right too. That's the design intent: partial credit for the right answer by the wrong reasoning shouldn't be possible, because the reasoning determines the answer.

Why I think this matters more than another leaderboard

I spend my time building agentic systems that are supposed to make judgment calls with minimal supervision — which model to route a task to, when a result is trustworthy enough to act on, when to escalate to a human instead. GeneBench-Pro is a clean instrument for exactly the failure mode I worry about most in that work: a model that produces a confident, well-formatted number without having earned it.

Three things stand out to me. First, a 31.5% ceiling on a research-taste benchmark is not a rounding error — it's a signal that scientific reasoning at this level is still substantially unsolved, whatever the surrounding hype around AI-accelerated science suggests. Second, the near-2x gap between first and second place tells you these questions are discriminating; this isn't a benchmark where every frontier model clusters near the top because it's actually testing recall in disguise. Third, and this is the part I'd flag to anyone evaluating models for scientific or analytical work: none of the case studies reward speed or fluency. They reward knowing which confound kills your estimate, and that's a much harder thing to train into a model than reading comprehension.

My informed take is that this kind of benchmark, not another chat leaderboard, is the one to watch if you want to know when AI systems are actually ready to be trusted with the parts of research that currently require a PhD's worth of accumulated skepticism. Right now, on the best available cross-lab measure, we aren't there. I'd rather have that number than the alternative — a demo that looks impressive and quietly gets the science wrong.