Why a single-cell foundation model stopped pretending genes have an order

I spend most of my time on generative and agentic systems built on text and code, so a genomics paper doesn't usually stop my scroll. Tabula did, because its authors made a decision I don't see enough teams make when porting the foundation-model recipe into a new domain: they looked hard at the actual shape of their data before deciding how to pretrain on it.

Tabula is a foundation model for single-cell RNA sequencing — the readout of which genes are active, and how much, in an individual cell. Two choices define it. First, it drops the assumption, inherited from NLP, that a cell's gene expression should be treated like a sentence. Second, it trains across institutions without those institutions ever handing over raw patient data. Both come from respecting the data's real structure instead of reusing whatever architecture worked last time.

Genes don't have a word order

Every prior single-cell foundation model I'm aware of — Geneformer, scGPT, tGPT — has to solve a problem that shouldn't exist: gene expression within a cell has no order, but a transformer pretrained the NLP way needs one. So each invents one. Geneformer ranks genes by expression to build a pseudo-sequence and does masked-language modeling on it; scGPT bins expression and uses its own attention scores to build a sequence for next-token prediction. Both treat an unordered measurement — a row in a cell-by-gene table, where columns are fixed genes — as if it had syntax.

Tabula stops pretending. It represents each cell as a permutation-invariant row of a table and swaps masked-language modeling for two objectives built for that structure. Gene-wise reconstruction corrupts 60% of a cell's gene values, drawing replacements from each gene's own expression distribution, and reconstructs the true values column by column. Cell-wise contrastive learning embeds both the original and corrupted views of every cell, pulling together embeddings of the same cell while pushing apart embeddings of different cells. Together they enforce consistent gene-level representations and stable cell identity, without imposing order on either axis.

Privacy as architecture, not policy

The second structural fact taken seriously: scRNA-seq isn't as anonymous as it looks — prior work has shown it's vulnerable to linking attacks that re-identify individuals across studies. That risk compounds as foundation models pool tens of millions of cells from thousands of patients across institutions, exactly the scale this class of model needs.

Tabula addresses this with federated learning rather than a data-use agreement. Each site — a hospital, an institute, a tissue bank — trains a client-specific embedder locally on its own private data, while a shared global transformer backbone is updated by FedAvg aggregation of transformer weights alone. Raw data never leaves the site that owns it; only weights and scalar metrics travel to the central server, which broadcasts the aggregated model back for the next round.

Chiron: the part most FL papers skip

Federated learning papers routinely stop at "we simulated N clients" in a notebook. This one ships the substrate. Chiron, built on the BioEngine distributed-compute layer, turns the federated loop into something an institution can stand up without a dedicated ML infrastructure team: a browser-based wizard points at a local dataset directory and emits a one-line launch command for Docker Compose, Podman, or Singularity/Apptainer. That starts two isolated containers — a data server that streams private data only within the worker's own network, and a GPU-bound trainer. A central orchestrator runs each round in four steps: broadcast global weights, train one local epoch per site, collect updated weights, FedAvg into a new global model.

Chiron's orchestrator isn't hard-wired to Tabula, either — it treats a trainer as a Flower-based client over BioEngine's RPC layer, so other single-cell models (the paper names scGPT and Geneformer) could join the same federation through the same contract. That's the difference between federated learning as a claim in a methods section and federated learning as reusable infrastructure.

Does it actually know biology, or just benchmarks

Beyond outperforming scGPT, Geneformer, and scFoundation on standard tasks — cell type annotation, batch correction, gene imputation, perturbation prediction — despite pretraining on fewer cells, the authors run a harder test: zero-shot recovery of gene regulatory networks in four developmental systems with regulatory logic validated by decades of biochemistry — hematopoiesis (11 transcription factors, 29 known edges), cardiogenesis (9 factors, 16 edges), neurogenesis (7 factors, 9 edges), and pancreatic endogenesis (8 factors, 22 edges).

The method: perturb a regulator gene in silico, track the resulting shift in a target gene over iterations, and classify the interaction as activating, repressing, or neutral from the direction of the shift — no fine-tuning. Accuracy holds above 0.83 across all four systems, and the model recovers 24 of 29 pairwise interactions in the core hematopoiesis network, correctly capturing how GATA1 and FLI1 jointly regulate KLF1. On harder combinatorial regulation — multiple regulators combining through something like a Boolean logic gate — performance is less uniform: 100% consistency on the simplest circuits, dropping to 12.5% on the most complex four-input gate. That drop-off is worth sitting with; it's a more honest signal than the headline number.

A concrete, falsifiable health claim

The paper's most ambitious test nominates candidate rejuvenation factors from human skin. The authors used matched young and aged fibroblast lines from three donors in the Baltimore Longitudinal Study of Aging, built an "age score" from genes tied to sterol homeostasis and protein transport, and an "identity score" from fibroblast marker genes. They embedded aged cells into a frozen Tabula model and iteratively optimized perturbations that push the reconstructed cell state toward a younger profile while keeping the identity score in fibroblast range. Ranking genes by their contribution across these paths surfaced three leading candidates: CPE, POSTN, and OLFM2.

Checked against mSALT, a mouse atlas of longevity interventions, these candidates showed up to 151% enrichment over random baseline for genes tied to lifespan-extending treatments. The authors then compared them against Yamanaka-factor partial reprogramming (OCT4, SOX2, KLF4) using a new inducible Perturb-seq construct on the same three lines. OCT4 or SOX2 alone lowered the age score; KLF4 raised it; all three lowered the identity score, exposing a real tradeoff between rejuvenation and identity. OCT4 and SOX2 did not move CPE or OLFM2 at all — Tabula's top candidates appear to sit on an axis orthogonal to reprogramming, not a rediscovery of known biology. A third candidate, mt-CO2, was induced by OCT4 and SOX2, showing partial convergence instead — a mixture of confirmation and independence more convincing than a clean sweep would have been.

What I'd actually watch

The durable idea here isn't any specific aging gene — those need wet-lab follow-up before anyone over-reads them. It's that privacy was designed into the training procedure rather than bolted on afterward, and the resulting model still had to prove it could do real biology zero-shot before anyone trusted it on an open question. The authors are upfront about the limits: all clients still share one global transformer body, with only a lightweight per-client embedder for tissue-specific variation, which caps how heterogeneous the federation can get before that shared backbone strains. The model is also transcriptomics-only for now, and 15 million pretraining cells is modest next to where this field is heading.

What makes the paper worth the read regardless is that the authors published both artifacts — the model and the federated training platform — rather than leaving the privacy claim as a paragraph nobody can independently verify. That's the harder thing to ship, and it's the part I'd bet generalizes furthest beyond single-cell biology.