What if a video model only looked at what actually moved

Every vision-language model I've put into production has the same blind spot: it treats a video stream like a stack of unrelated photographs. Sample a frame, chop it into patches, tokenize every patch, feed the whole grid to the transformer, repeat for the next frame. It works, but it is spending compute on pixels that haven't changed since the last frame — the same wall, the same sky, the same static background — over and over, dozens of times a second. A new paper, Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model, attacks that waste directly, and the fix is one of those ideas that feels obvious in hindsight: video codecs already solved this problem decades ago, so borrow their solution instead of re-deriving it.

The paradox this is responding to

The authors frame the motivation around Moravec's paradox: VLMs are excellent at slow, effortful reasoning over a single complex image, and comparatively bad at the kind of fast, cheap perception a person does without thinking — noticing that something in a video just changed. Standard architectures make this worse by processing streaming video as if it were a pile of independent stills. There's no notion of "this region is identical to a moment ago," so the model re-encodes it anyway. For offline captioning that inefficiency is annoying. For real-time streaming perception — a camera feed, a robot's view, a live video call — it's the difference between a system that keeps up and one that doesn't.

Reusing the codec instead of re-inventing it

The core contribution is a tokenizer called Mage-ViT, and its trick is to stop treating incoming video as raw pixels and start treating it as what it already is on disk or on the wire: compressed video with motion vectors and residual energy attached. Compressed streams already separate anchor (I) frames, which are encoded in full, from predicted (P) frames, which only store how blocks moved and what changed relative to the previous frame. Mage-ViT reads those signals at a 16x16 patch level and uses them to decide, patch by patch, whether a region is dynamic and entropy-rich enough to deserve a full token, or static enough to skip. A static hallway gets encoded once and then left alone; a hand reaching for a cup gets tokens spent on it every frame it's moving.

That's the mechanism behind the headline number: over 75% fewer visual tokens, while the paper reports spatiotemporal context is preserved rather than degraded. Fewer tokens per frame means shorter sequences for the language model backbone to attend over, which is where the reported up to 3.5x wall-clock inference speedup comes from — it's not a smaller model doing less work, it's the same model doing work on a much smaller, more relevant set of inputs.

A second brain for when to react

The efficiency gain from Mage-ViT is paired with an architectural choice that mirrors it: a bio-inspired dual-system design. A lightweight System 1 acts as an event gate, cheaply watching the stream for something worth paying attention to. A heavier, causal System 2 decoder only engages when System 1 flags a moment that matters. It's the same instinct as the tokenizer applied one level up the stack — don't spend the expensive reasoning pass on every tick of the clock, spend it when there's actually something to reason about. Together, gate and decoder are what let Mage-VL do proactive streaming perception rather than waiting to be asked a question about a frame.

What the numbers say about the tradeoff

The part I find most convincing isn't the speedup, it's that the speedup doesn't cost accuracy. Mage-ViT was trained from scratch on roughly 560M unlabeled images and 100M unlabeled video frames, and the paper reports it matches or outperforms flagship encoders trained on billions of image-text pairs — a data-efficiency claim that's easy to make and hard to back up, but here it's backed by the downstream model's results. Mage-VL-4B matches Qwen3-VL-4B on static image tasks, so nothing was sacrificed on the workload the field already optimizes for, while showing clear gains on video understanding and 2D/3D spatial reasoning — the tasks that actually require tracking change over time. It also comprehensively surpasses Phi-4-reasoning-vision at 15B, which matters less as a leaderboard line and more as evidence that the token budget you save by being selective can be reinvested in reasoning quality rather than just spent on inference savings.

The authors also report seven empirical findings alongside the model itself — on pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and a "Zero-Vision SFT" recipe for multimodal RL. That's a lot of ground, and worth reading the paper directly if you're building training recipes rather than just consuming the model.

Where I think this goes

What I take from this paper is bigger than the specific 75%/3.5x numbers. It's a reminder that a lot of the compute we throw at video understanding is compute we've already paid for once, upstream, in the codec. Motion vectors and residual energy exist in every compressed video stream whether or not a model ever looks at them. Treating them as a free, pre-computed saliency signal rather than re-deriving "what changed" from raw pixels is the kind of systems-level efficiency that tends to compound: it doesn't just make training cheaper, it changes what's feasible to run in real time on constrained hardware — a robot's onboard compute, an edge device, a live agent watching a screen. The open question I'd want answered next is how well this generalizes past clean, well-compressed input: real-world streaming pipelines involve re-encoding, variable bitrates, and codecs that aren't tuned for downstream model consumption, and I'd want to see how sensitive the token-selection signal is to that kind of noise before treating codec-native tokenization as a drop-in replacement for uniform sampling everywhere.