Self-driving datasets: when the agent out-curates the database

The data was always the ceiling

In scientific machine learning, the model is rarely the thing holding you back. The dataset is. A toxicity classifier, a binding-affinity predictor, a permeability model — each inherits the scale, the coverage, and the error rate of the curated table it was trained on. We have spent decades and enormous sums building those tables: ChEMBL, the Therapeutics Data Commons, ClinVar, the Human Protein Atlas. They are real assets. They are also expensive to maintain, they lag the primary literature by years, and they discard most of the context that produced each number.

A recent paper out of Penn, Self-Driving Datasets: From 20 Million Papers to Nuanced Biomedical Knowledge at Scale, argues we no longer have to accept that trade. PubMed itself — the unstructured primary literature — can be turned into structured datasets that are larger, more accurate, and richer in context than the curated databases they would replace. I find the result interesting less for the biology than for what it says about building agentic systems that produce data rather than prose.

Three things wrong with curated data

The paper's framing of the problem is worth stating plainly, because it applies well beyond biomedicine.

Curated data lacks nuance. Most properties worth predicting are not functions of the molecule alone. Oral bioavailability depends on whether the drug is taken fed or fasted, on its formulation, on the population it was dosed in. A reaction yield depends on procedure and conditions. A model that sees only the molecule can, at best, learn the marginal distribution over all those hidden conditions. Text makes the conditions first-class; a table flattens them away.

Curated data is incomplete. Repositories lag the literature, and even at steady state they capture a small fraction of what has been measured. Worse, the same phenomenon gets reported in incompatible forms — hepatotoxicity as enzyme binding in one paper, as a population-level adverse-event signal in another. Awkward to force into one schema, easy to state in language.

Curated data is less accurate than we assume. When the authors audited widely used benchmarks against human judgment, they measured 16.5% label error on the TDC blood-brain-barrier set and 7.3% on TDC oral bioavailability. A model trained on those labels inherits the error as a noise floor and a ceiling on achievable test performance. You cannot validate your way past a corrupt gold standard.

What "self-driving" actually means

The system has three layers, and the order matters.

First, a corpus: 22.5 million licensed PubMed papers, roughly 2.5 trillion tokens, with text and tables extracted to markdown. Second, an entity-tagging model — fine-tuned from gpt-oss-20b — that tags 4.5 billion entities across 19 biomedical categories, grounded in nine reference ontologies and, for small molecules and proteins, mapped directly to SMILES strings and amino-acid sequences. Third, a hybrid sparse–dense retrieval layer exposing entity-filtered semantic queries over the tagged corpus. One design choice stands out: the embeddings re-rank candidate results rather than filter the corpus, so recall is preserved and precision becomes a tunable knob rather than an accident of the embedding.

On top sits Starling, a multi-agent system that takes a natural-language task — "find every paper reporting the oral bioavailability of a small molecule" — and autonomously designs the retrieval filters, induces an extraction schema, and emits structured records, each linked to the supporting passage it came from.

The design inversion is the part I keep coming back to. A general-purpose deep-research agent reads a handful of relevant documents and synthesizes a narrative. Starling does the opposite: it exhaustively sweeps the corpus for every instance of a phenomenon and emits one structured record each time. Report-writer versus census-taker. Most agent demos are the former; the valuable industrial work is almost always the latter.

The pipeline: structure is built into the substrate before the agent ever reasons over it.

The accuracy inversion

Across six therapeutic tasks, Starling produced roughly 6.3 million judge-filtered records, with per-task scale ranging from 91K to 3M. The scale is what you would expect: 31K blood-brain-barrier molecules against B3DB's 8K; 19K oral-bioavailability molecules against TDC's 600; three million gene-disease extractions against ClinVar's 17.7K associations.

The surprise is accuracy. Frontier-model rejection of the kept extractions runs 0.6–7.7% across tasks. On a head-to-head human audit, Starling's error rate came in below the curated benchmark on the same property — 5.9% versus 16.5% on blood-brain barrier, 5.1% versus 7.3% on oral bioavailability. (On LD50 toxicity the two were effectively tied.) The automated pipeline is, on the properties that matter most here, more accurate than the hand-curated gold standard.

More records and a lower audited error rate than the databases it would replace.

The reason is mundane once you see it: the curated label discarded the context that explains the disagreement. An auditor holding the supporting passage can adjudicate a contested value; an auditor holding a bare row cannot.

Nuance is the real prize

Scale and accuracy are the headline, but the supporting passage is the lasting contribution. For a single molecule, the residual variation in oral bioavailability is often explained by experimental context — fed versus fasted dosing, a drug interaction, the formulation. A tabular row collapses that to one number and calls it the property. The passage keeps the conditions, which means a downstream model can condition on them instead of marginalizing over them. That is the precondition for the multimodal predictive and generative models the field actually wants.

What I take from it

A few lessons travel directly to production agentic systems, biomedical or not.

Ground the agent in structure before you let it reason. The tagging model, the ontologies, and the retrieval layer are what make Starling's job tractable; an agent loop over raw text would not. Most of the engineering lives in the substrate, not the prompt.

Separate recall from precision and let the agent tune them. Re-ranking over a recall-first candidate set beats a hard embedding filter whenever you care about coverage.

Audit the output the way you would audit the thing it replaces — and at a stated cost. Here it was about a tenth of a cent per extraction. Measure it, and the comparison becomes honest.

And treat the human gold standard as a hypothesis, not as truth. When a well-built system disagrees with the benchmark, the benchmark is sometimes the one that is wrong. The authors release the agent, the tagging model, and all six datasets, which is the right move: the most useful thing self-driving datasets may do is not replace curation but re-open it — turning a dataset from a static artifact that decays into a query you re-run as the literature grows.