Seeing is not predicting: what SpaceCast-Bench says about spatial reasoning in VLMs

Most spatial benchmarks for vision-language models ask a question whose answer is already in the pixels: is the chair left of the table, how far is the lamp from the door. A new paper from Zhejiang University, SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models, asks something harder. The model sees a scene, is told about a change it never observes, and has to answer a question about the scene after the change.

That is the gap between perception and prediction, and it is the gap that matters for anything that acts in the physical world. The authors' example is a good one: a robot retrieving an envelope from under a book with a cup on top has to predict that sliding the book will move the cup. Reading relations off an image is not enough. The model needs a representation of the scene it can update.

Observe, transform, infer

The benchmark has three steps. The model receives an ordered sequence of RGB views of the initial scene, a natural-language transformation, and a spatial query. It has to rebuild the relevant part of the scene from the views, apply the transformation, and evaluate the query in a result state it never sees.

Tasks sit at three levels. L1, static perception, uses the identity transformation, which gives a perceptual baseline. L2, local prediction, moves, orbits, or removes a single object, together with anything attached to it. L3, global prediction, rotates the whole layout, so the model has to tell a rotated scene apart from a moved camera. The queries cover direction, distance, visibility and occlusion, and attachment. Directions are asked in camera-centric, object-centric, and world-centric frames.

The benchmark has 3,862 questions across 182 real scenes from ScanNet and ScanNet++, split into 16 task types. What I like most is how the labels are made. They are not written by annotators guessing at outcomes. The pipeline computes them from 3D geometry: it applies the transformation to the meshes and reruns a relation engine on the new state. It drops cases near decision boundaries. Ten human annotators then checked that each question is answerable from what the model actually sees, with 98.9% pairwise agreement on single-answer questions before adjudication. A benchmark is only as good as its labels, and programmatic labels from posed RGB-D scans are about as good as this kind of task gets.

The view chains are also built with care. Cross-view questions use two anchor views that each show one group of objects and hide the other. "Bridge views" along a depth-verified corridor link the two anchors. The model cannot answer by looking at a single frame. It has to stitch views together.

The gap

The authors evaluated 21 models: nine proprietary, six general-purpose open-weight, and six built specifically for spatial reasoning. The headline number is the gap. Gemini 3.5 Flash leads at 58.0%, with Gemini 3 Flash at 56.7% and Qwen3.7-Plus at 55.3%. Humans score 87.2%. The best open-weight model, Qwen3.5-27B, reaches 39.3%.

The most uncomfortable result is about the spatial specialists. They score 25.7% to 29.1%, below every general-purpose model and close to the 25.9% random baseline. The authors read this as evidence that current spatial fine-tuning does not carry over to updating a spatial state. On attachment questions, SenseNova-SI-1.2 gets 13.4% and SpatialLadder 9.3%, while Qwen3.7-Plus gets 80.4%. Training on a narrow slice of spatial perception seems to produce narrow skills.

The level breakdown makes the point sharper. Strong proprietary models fall by around 20 points from L1 to L2: Gemini 3.5 Flash drops 21.8 pp and Qwen3.7-Plus 20.6 pp. They then recover on L3 global rotations, where a rough sense of the overall layout seems to be enough. Doing well on static perception does not mean a model can update its picture of the scene after an edit. Object-centric directions are the weakest frame for general models. A model reasoning from the point of view of an object in the scene, not the camera, is still unreliable.

Two more findings stand out. Chain-of-thought prompting gives no consistent benefit: it helps some models at some levels and hurts others. Removing bridge views lowers directional accuracy in every reference frame. The largest drop is for world-centric direction, from 39.9% to 34.6%. Models need the connecting views to put together a scene-level representation.

Explicit 3D beats imagined outcomes

This is the result I would bring to anyone building embodied or spatial agents right now. A popular idea is that generative world models can help reasoning by showing the model the future: render the scene after the intervention and let the VLM look. The authors test this directly against the simpler option of giving the model explicit 3D structure.

Explicit 3D evidence wins clearly. Bird's-eye-view images help both proprietary and open-weight models. Textual 3D metadata helps much more: +30.7 pp for proprietary models and +10.9 pp for open-weight ones. Outcomes generated with Qwen-Image-Edit, GPT-image-2, or Seedance2-mini cause smaller and less consistent changes. A manual audit of 600 local-intervention questions explains part of this. Among wrong answers, 80.3% under Qwen-Image-Edit and 38.3% under GPT-image-2 were linked to generation failures: artifacts, missing objects, deformation, and scale or perspective mismatch.

My reading as an engineer is that, for strong general models, the main bottleneck is recovering 3D structure from pixels, not reasoning over it. Once the geometry is given to them, they can apply the transformation and reason about relations well. That matches what I see in production agent systems. When a model has trouble with a structured problem, the reliable fix is usually to give it the structure in a form it can read, not to add another generative stage that brings its own failure modes. A generated image of the future is one more noisy observation that still has to be lined up with the original views. A list of object positions is not.

The specialists, notably, decline with 3D metadata (-3.3 pp). Fine-tuning has made them brittle to representations they did not see in training.

Training on the pipeline

The same pipeline produces training data from scenes kept separate from the benchmark. Staged SFT on Qwen3-VL-4B, with L1 first and then L2/L3 with L1 replay, raises overall accuracy from 34.0% to 65.7%. That is above the best proprietary model on this benchmark. It also lifts the macro average across six out-of-domain benchmarks from 33.5% to 38.1%, with the largest gain on DSI-Bench (+10.2 pp). GRPO reached only 37.7% in-domain. The authors suggest the tasks are too hard for on-policy training to get consistent reward.

I would not read too much into a 4B model beating frontier models in-domain. That is what benchmark-aligned training does. The out-of-domain gains are the more honest signal, and they are real but modest.

What I take from it

SpaceCast-Bench separates seeing a scene from predicting how it changes, in a clean and diagnosable way, and today's models are much better at the first. If you are building agents that manipulate or plan in physical space, three practical points follow. Don't assume a strong perception score means the model can predict. Give the model explicit geometry when you have it. Be skeptical of generated future frames as a reasoning aid until generation fidelity and cross-view alignment both improve. The authors say plainly in their ethics statement that these models are not ready for safety-critical embodied deployment, and the numbers support that.

The dataset is on Hugging Face and the code is on GitHub.