Whole-body foundation models and the dominant-organ shortcut
Pan-FM is a foundation model trained across seven organs at once and built to tolerate missing scans. Its core idea, saliency-guided masking, uses the model's own attention to stop it from leaning on a few dominant organs and force balanced, whole-body learning.
Most foundation models in medical imaging are specialists. They learn from one organ in one modality — brain MRI, or cardiac cine — and they become very good at that narrow slice. But aging and disease do not respect organ boundaries. Metabolic dysfunction registers in the liver, pancreas, and adipose tissue together; cardiovascular risk is written across the heart and kidneys at once. If you want a model that reflects how the body actually fails, it has to read more than one organ.
Pan-FM, a recent paper from Wu, McIlvain, Yu, and Wen, builds exactly that — a pan-organ foundation model pre-trained across seven organs. What I find useful about the work is not the ambition but the honesty: the moment you train one model on many organs, two failure modes appear, and the paper is mostly about naming and fixing them.
The data is missing, and not at random
The first problem is structural. In any real cohort, not every subject has every scan. Protocols differ, sequences fail, follow-ups get skipped. So a whole-body model has to cope with organs that are simply absent for a given person.
The harder version of this is that the missingness is not random. Whether a scan exists often correlates with the very thing you are trying to predict — sicker subjects get imaged differently, certain sites run certain protocols. If you handle this by dropping incomplete records, you throw away statistical power, narrow your population, and bake the correlation into the model as bias. Missingness has to be treated as a property of the data the model lives with, not a preprocessing step you clean away before training.
One backbone, seven organs
Pan-FM's answer is a single unified backbone that ingests imaging from brain, heart, adipose, liver, kidney, spleen, and pancreas, and is built to handle organ missingness during both training and inference. It is pre-trained with masking-based self-distillation: a student network sees a masked view of the input and is trained to match a teacher network's representation of the full view. This is the same self-supervised recipe that has worked well in single-domain imaging, extended to a set of organs rather than a single one.
Handling missingness during both phases means the same model that learns from a subject with all seven organs can still produce a representation for a subject with only four. That is the property you need to deploy against real, incomplete data.
The shortcut nobody asked for
The second problem is subtler, and it is the one I keep seeing in multimodal systems outside medicine too. When you train a single model on several inputs, it does not weight them evenly. It finds whichever inputs most cheaply reduce the loss and leans on those, leaving the rest under-learned.
In Pan-FM, naive multimodal pre-training produced exactly this: a dominant-organ shortcut, where the model over-relied on a couple of organs — adipose and heart — and learned thin representations for everything else. On the aggregate pre-training objective this looks fine. In practice it is fragile. Remove the dominant organ at inference and performance falls off; any disease that lives mainly in an under-attended organ is poorly served. This is the multimodal version of shortcut learning, and it is the default outcome, not an edge case.
Saliency-guided masking
The fix the authors propose is the part worth remembering. Saliency-Guided Masking (SGM) reads the model's own attention distribution to see which organs it is leaning on, then adaptively masks those dominant organs during pre-training. By hiding the easy signal, it forces the model to extract information from the organs it would otherwise coast past.
I like this because it is the modality analogue of hard-example mining: instead of up-weighting the samples a model gets wrong, you mask the inputs it gets right too easily. The masking signal is the model's own confidence, so it needs no extra labels and adds negligible compute. It also drops into existing self-supervised frameworks without rearchitecting them — which is the difference between a clever idea and one people will actually adopt.
What the results show
Evaluated on the UK Biobank, Pan-FM predicts more accurately than both single-organ and multi-organ baselines across 13 disease categories and 14 single disease entities. The number I weight most is the robustness one: it holds up better than the baselines when organs are missing at inference. A model that degrades gracefully as inputs drop out is the one you can run against the messy cohorts that exist, rather than the complete ones that do not.
I would not over-read a single result. This is one cohort, imaging only, and UK Biobank is a relatively healthy, homogeneous population — the missing-not-at-random problem is almost certainly worse in clinical data drawn from routine care. The claim the paper earns is narrower and still useful: across seven organs, balanced pre-training plus explicit missingness handling beats the obvious baselines.
What carries over
Two ideas here generalize well beyond medical imaging, and they are why I bothered writing this up.
First, missingness is a design constraint, not a cleaning task. If your data is incomplete in the wild, the model should be trained and queried under that same incompleteness — anything else trains a model for a world that will not show up at inference.
Second, shortcut learning is the quiet default of multimodal pre-training, and the model's own attention is a cheap instrument for fighting it. The Pan-FM recipe — find what the model over-attends to, mask it, repeat — is not specific to organs. Any multimodal system where one stream dominates the loss is a candidate for the same correction.