A New Benchmark for Spatial Reasoning: Can VLMs Imagine How Space Changes After an Object Moves?

Imagine that in front of you are three photos of the same indoor scene, taken from different angles. A floor lamp is originally located beside a sofa. Now, please answer a question that does not appear directly in any of the input photos: if the lamp is moved 2 meters toward the camera, in what direction and at what distance will it be relative to the coffee table?

For humans, we can usually combine the three photos, roughly reconstruct the 3D layout of the room, update the position of the floor lamp in our minds, and judge its new relationship with the coffee table. For vision-language models (VLMs), this type of question cannot be answered by directly looking up pixel relationships in a certain image. The model needs to integrate multi-view information, form a judgment of the scene structure, perform the hypothetical object movement, and re-derive the edited spatial relationship.

MindEdit-Bench is built around exactly this capability: can VLMs reason about the spatial consequences caused by object edits? Specifically, when an object is translated or rotated, how will its relative position, distance, and even occlusion relationship with surrounding objects change? As a community focused on data, evaluation, and open collaboration, ZODA was deeply involved in this research. The paper has now been released on arXiv, and the dataset has been open-sourced on Hugging Face.

1. A Gap in Spatial Evaluation: Reasoning After Object Movement

Before MindEdit-Bench, most spatial understanding evaluations stayed at the level of observational reasoning: models only needed to identify relationships already present in the input images, such as “is the cup to the left or right of the table?” or “which of the two objects is closer to the other?” As for evaluations involving “What-If” reasoning, they generally fall into two categories:

• Observer-level: the camera or person changes position, while the scene itself remains unchanged. Representative benchmarks include SQA3D and MindCube.

• Image-editing: pixels are directly modified, with the focus on checking whether the generated image is natural and whether the target area has been correctly changed. Spatial relationship reasoning is not the main evaluation goal. Representative work includes SpatialEdit.

Neither of these two paradigms touches on a more fundamental question: objects can move independently of the observer, and the spatial relationships in the scene will be reconstructed accordingly. Whether models can answer “what will happen after an object is moved or rotated” is still a gap in existing evaluations.

To our knowledge, MindEdit-Bench is the first to push evaluation toward object-level counterfactual spatial reasoning. It requires models to answer reasoning questions about “how spatial relationships will change after an object moves or rotates,” based on provided multi-view scene images and a textual description of a hypothetical edit operation.

The reason this task forms a real reasoning challenge lies in one core constraint: the edited object configuration never appears in any input image. The model cannot directly obtain the answer by retrieving the pixel positions of neighboring objects in a 2D image. Instead, it must judge a spatial state that is described by text conditions but not directly shown in the images. In this way, the benchmark moves evaluation from “what is seen” to “what would happen if...”

2. Tasks and Data Construction

MindEdit-Bench sets up six graded tasks around perception, viewpoint transformation, and counterfactual editing. It uses three smartphone photos of the same scene as input, reconstructs an object-level 3D scene graph through a feed-forward pipeline, and automatically generates standardized questions.

2.1 Six Types of Spatial Reasoning Tasks The six tasks are divided into three groups: perception, viewpoint transformation, and counterfactual editing.

First group: Perception

• L1 Cross-view object relationship: judge which pair of objects distributed across different views has the closest 3D surface distance.

• L2 Camera motion: judge what translation and rotation the camera undergoes from one view to another.

This group of tasks mainly describes the already observed scene and also forms the foundation for subsequent tasks.

Second group: Viewpoint transformation

• L3a Single-view viewpoint transformation: imagine standing at the position of object A and facing object B, then judge the direction and distance of object C.

• L3b Cross-view viewpoint transformation: the task format is the same, but the related objects are distributed across different photos.

The model cannot directly apply the “left,” “right,” “front,” and “back” in the image to the answer. It must first switch to a new spatial reference frame. L3b is harder than L3a because the reference objects are not in the same image. Cross-image association itself is a hurdle.

Third group: Counterfactual editing

• L4 Spatial editing: translate an object, or rotate it around another object, and then judge the edited direction and distance.

• L5 Cross-view visibility editing: judge which operation can make an object appear or disappear in another view.

L4 and L5 are the core innovations of MindEdit-Bench. The scene state corresponding to the correct answer has never appeared in any input image. The model cannot directly guess correctly by relying on visual memory or 2D cues.

Each question is set with 8-24 candidate answers. The options are systematically generated according to direction, distance, motion axis, and operation type, reducing random-guess accuracy to about 4%-13%. This design can also locate errors to specific dimensions: even if the model ultimately outputs only one letter, we can determine whether it is wrong on direction, distance, rotation operation, or whether it directly chooses “none of the above” on difficult questions.

2.2 Benchmark Data Construction

High-quality 3D data usually relies on professional scanning devices, dense videos, or per-scene optimization, with high collection and computation costs. MindEdit-Bench chooses a lighter route: each input group only requires three smartphone photos of the same indoor scene. The dataset covers 120 self-collected indoor scenes that had not previously entered public datasets. The photos are processed through a feed-forward pipeline to form object-level 3D scene graphs, and candidate questions are then automatically generated from the scene graphs.

The pipeline is divided into two stages, progressively ensuring data quality:

• Stage 1: Per-view understanding. The system first generates an object vocabulary for each scene, then estimates the depth and camera position of each photo through Depth Anything 3 (DA3), and extracts segmentation regions for objects such as sofas, tables, and lamps through SAM3.

• Stage 2: Cross-view 3D association. The object regions in each image are back-projected into 3D space. Based on volume overlap, containment relationships, and center distance, the system judges which regions in different photos belong to the same object, and uses horizontal structures such as floors, ceilings, and tabletops to recover the gravity direction. The final scene graph records object positions, distances, and cross-view visibility, providing geometric basis for automatic question generation.

Human review. After per-view understanding and cross-view 3D association, human review is introduced to further verify the generated questions. The automated stages produced 1,172 candidate questions, which were then manually reviewed. Among them, 758 question stems and answers passed review directly, 245 had clear question stems but answers corrected by humans, and 169 were discarded due to issues such as referential ambiguity, ultimately forming 1,003 human-verified questions.

3. Evaluation Results: The Strongest VLMs Are Still Far Below Human Level

The evaluation covers 15 mainstream VLMs, including GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4, Qwen3-VL, and others, and compares them with a human baseline from three professional annotators. All models complete the evaluation in a zero-shot manner. Each model is run three times, and the main result is calculated only according to the final answer letter.

3.1 Overall Performance: A 53.3 Percentage Point Human-Model Gap The results are shown in Figure 6. Across the six task types, the average accuracy of the 15 models ranges from 8.3% to 30.6%. Across all 1,003 questions, the overall average accuracy of the models is only 18.0%. Among them, Gemini-3.1-Pro-Preview ranks first with 34.6%, and Qwen3.6-Plus ranks second with 31.1%.

In comparison, the majority vote of the three annotators reaches 80.8%-97.0% accuracy across the six task types, with an overall accuracy of 87.9%, 53.3 percentage points higher than the strongest model. Humans can answer stably under the same photos, questions, and options, showing that these tasks are clearly solvable. The significant human-model gap further indicates that current VLMs still have obvious shortcomings in constructing, maintaining, and operating 3D spatial relationships.

3.2 Failure Analysis: Candidate Answer Design Reveals Where Models Go Wrong

The structured design of candidate answers makes three typical failure modes visible:

• Weak depth-direction perception, relatively reliable left-right direction. In L4, when the answer lies in a pure depth direction, the model’s average accuracy is only 7.4%; for pure lateral direction it is 18.0%, and for diagonal direction it is 24.3%. When models judge the direction incorrectly, they are more likely to land on an adjacent 45° direction and rarely land on the completely opposite direction. This shows that models can often locate only a rough area and have difficulty stably recovering depth relationships in the camera’s front-back direction.

• Rotation reasoning is significantly harder than translation reasoning In L4, the average accuracy for translation questions is 21.8%, while rotation questions reach only 13.5%; in L5, there is also a gap of 8.3 percentage points. Translation mainly updates position along a fixed direction, while rotation requires re-deriving angles and relative relationships around a reference point. The more complex the operation chain, the easier it is for the model to lose its stable representation of the scene.

• Difficult tasks cause models to show “avoidance” behavior L5 contains a “none of the above” option. This option accounts for only 14.2% of the true answers, but some weaker models choose it more than 70% of the time, with the highest reaching 95.1%; other models never choose this option at all. When models cannot complete the continuous reasoning of “edit object - project to another view - judge occlusion,” the results show two extreme strategies: over-relying on the fallback option, or completely ignoring it (Figure 8).

To test whether explicit reasoning prompts can alleviate the above failures, the evaluation further adds a general chain-of-thought (CoT) prompt to four representative models, asking the models to first explain step by step and then give the answer. GPT-5.4’s overall accuracy improves by 12.8 percentage points, and Qwen3-VL-32B improves by 6.0 percentage points; the highest score of the entire benchmark only rises from 34.6% to 35.8%, still about 52 percentage points away from humans. Step-by-step prompting can help some models organize reasoning, but it is still insufficient to bridge the capability gap. The real challenge lies in forming reliable spatial judgments from limited views, and maintaining and updating those judgments through continuous operations.

4. ZODA’s Perspective: Spatial Understanding Supports Embodied Intelligence and World Models

Spatial understanding is becoming a key foundational capability for multimodal AI as it moves toward embodied intelligence and world models. For models to plan actions, predict state changes, and interact with objects in real environments, they need to continuously maintain 3D scenes and reason about the chain changes brought by movement, rotation, and occlusion. MindEdit-Bench shows that current VLMs still have a significant gap from humans in this type of capability, and also points to more specific breakthrough directions for future data construction and model training.

As an open data and evaluation community, ZODA hopes to connect data production, benchmark design, model evaluation, and real applications, so that progress on key capabilities has verifiable tasks, data, and results as its basis. In the future, ZODA will continue to promote the construction of datasets and evaluations related to spatial intelligence, embodied intelligence, and world models, and work with research institutions, developers, and industry partners to improve an open and reproducible evaluation ecosystem.

5. Conclusion: From Recognizing Space to Reasoning in Space

Spatial intelligence includes recognizing object positions, and also includes reasoning about spatial changes. Models need to move away from reliance on local 2D cues and gradually form, maintain, and update more stable spatial judgments.

MindEdit-Bench breaks this capability down into six types of reproducible tasks, and through experiments directly shows that there is still a clear gap between current VLMs and humans. These results make failures that were previously difficult to define measurable and discussable, and also provide clearer directions for subsequent model training, spatial tools, and agent systems.

ZODA will continue to advance open collaboration in AI data and evaluation. Follow ZODA to learn about the latest progress in spatial intelligence, embodied intelligence, and model evaluation. We also welcome related teams to participate in exchange and co-building.