Data mixing, not filtering, is what actually improves VLM pretraining data

Vision-language model training has a data problem that's different from the one language models have. We spent years learning that curation for LLM pretraining is mostly about filtering: throw out low-quality web text, dedupe aggressively, keep what survives. VLM data is messier -- captions, interleaved image-text documents, plain text, and instruction-tuning examples all compete for the same token budget -- and until now nobody had a controlled way to test which curation moves actually help. A team from the group behind DataComp and DataComp-LM has now built that benchmark. The paper is DataComp-VLM: Improved Open Datasets for Vision-Language Models, and the result worth sitting with is this: the lever that moves quality is mixing, not filtering.

A benchmark built to isolate the variable that matters

DataComp-VLM (DCVLM) is a controlled experimental harness, not just a dataset release. The authors assembled 160 source datasets spanning four distinct data types -- image-caption pairs, multimodal interleaved documents, text-only corpora, and instruction-tuning data -- into a pool of 6 trillion multimodal tokens. Participants (in this case, the authors themselves, running the first large sweep) can test curation strategies -- filtering, mixing, formatting, sampling -- while holding everything else fixed: model scale (1B to 8B parameters), token budget (6.25B to 200B tokens), and architecture. Trained models are then scored on up to 52 downstream benchmarks spanning 9 domains, which is enough coverage to catch a curation strategy that helps on one axis while quietly hurting another.

That design matters because most VLM data papers report a single new dataset and a single new score, with the curation recipe and the eval suite both chosen by whoever built the dataset. DCVLM decouples the two: it's a fixed testbed you can run many recipes through and compare on equal footing, which is exactly what DataComp did for image-text pairs and DataComp-LM did for text. This is the same lineage applied to the harder, multi-source VLM setting.

The finding: mixing beats filtering

Across their sweep, the authors found that the composition of the training mixture -- what fraction comes from captions versus interleaved documents versus text-only versus instruction data -- drives downstream quality far more than how aggressively any individual source is filtered. This is worth pausing on, because filtering is where most curation effort in this space has historically gone: quality classifiers, dedup, safety and NSFW filters, CLIP-score thresholds on caption pairs. Those things aren't worthless, but per DCVLM's results they're a smaller lever than getting the ratio of data types right.

Instruction data wins as you scale, and the gap widens

The more specific and more actionable finding is about which mixtures win: instruction-heavy mixtures scale better than caption-heavy ones, and the gap between them widens as you move to larger token budgets. In other words, a recipe that looks fine at 6.25B tokens can be a worse recipe at 200B tokens if it's too caption-heavy -- the ranking of mixtures isn't stable across scale, and betting on caption-dominant data (the traditional VLM pretraining diet, going back to CLIP-style contrastive pairs and BLIP-style caption corpora) is a bet that gets worse the more compute you throw at it. If you're planning a training run at the top of your budget, the recipe you'd pick from a small-scale ablation may not be the recipe that wins at full scale.

The resulting baseline

Applying what they learned, the authors built DCVLM-Baseline, a dataset assembled from the best mixture found in their sweep. Training an 8B model on it for 200B tokens reaches 63.6% accuracy on their 33-task core suite. The comparison point is FineVision, currently the strongest open VLM training dataset publicly available -- DCVLM-Baseline beats it by +5.4 percentage points at the same scale. That's a meaningful jump for a result that comes purely from rearranging what's already public data, with no new labeling or proprietary sources. The authors state that DCVLM and its accompanying artifacts -- the benchmark harness, the mixtures, and presumably DCVLM-Baseline itself -- will be released publicly, though the paper doesn't give a firm date.

Why this matters if you're building VLM pipelines

If you're assembling pretraining data for a vision-language model today, the practical takeaway is to spend your engineering effort on the mixture ratio before you spend it on filter quality. Concretely: don't default to a caption-pair-dominant corpus because that's the historical convention, budget real weight for instruction-tuning data in the pretraining mix itself rather than treating it as a late-stage fine-tuning step, and re-validate your mixture choice at (or close to) the token budget you'll actually train at, since a ratio that wins small can lose big. It's also a reminder that the DataComp lineage's real contribution isn't any single dataset -- it's the discipline of building a benchmark that lets you tell the difference between a curation idea that sounds good and one that's actually measured to help.