Beyond Trial and Error: SolveEdit Benchmarks Visual Problem Solving in Generative AI

Editing an image with generative AI can feel like a lottery: the small change you want arrives with changes you never asked for. Anyone who has used AI for precise image editing will recognize the frustration. You ask the model to move a cup slightly to the left. It moves the cup, but also changes the lighting, alters the background texture, or subtly reshapes a person’s face.

The challenge goes beyond rendering quality. A successful edit requires the model to determine what should change, how it should change, and what must remain intact.

To study this capability, ZODA, together with researchers from the Hong Kong University of Science and Technology, the University of Chinese Academy of Sciences, Zhejiang University, and the University of Tokyo, introduces SolveEdit. The benchmark contains 2,728 cases that test visual problem solving through scene transformation. The team also introduces SolveEdit-Plan, a method that determines the intended transformation before passing an instruction to the generator.

1. The Challenge: Rendering a Scene Is Only Part of the Task

Developers and creators need edits that achieve a specific goal while preserving the surrounding scene. Yet a locally correct change can still produce an unusable result if the model also alters unrelated objects, textures, or spatial relationships.

This makes precise editing costly. Users may need repeated attempts to obtain an output that both completes the task and preserves the details they care about.

There is another difficulty: many real-world requests describe an outcome without specifying the operations needed to achieve it. In these cases, the model must first work out a valid transformation from the image and the goal.

Consider circuit repair. “Connect P2 and P3 with a yellow wire” specifies the operation. The model needs to locate the terminals and draw the connection. “Fix the broken circuit” leaves the operation unresolved. The model must inspect the circuit, identify the fault, determine a valid repair, and execute it without disturbing working components.

SolveEdit focuses on this question: Can a generative model understand a visual scene, determine a valid solution, and carry it out without detailed editing instructions? A convincing image is not necessarily a correct answer. A model may produce a polished result while violating the rules, logic, or spatial constraints of the task.

2. The Approach: Visual Problem Solving Under Explicit Constraints

SolveEdit provides a framework for measuring whether models can determine and execute valid scene transformations.

2.1 A Benchmark of 2,728 Cases and a Contract-Based Evaluation Framework

SolveEdit contains 2,728 cases across 10 domains and 54 subdomains, covering tasks such as arranging objects, planning routes, correcting numerical content, and repairing structural relationships.

To account for the fact that visual problems can have multiple valid solutions, the team introduced SolveScore, a reference-free scoring mechanism, replacing traditional pixel-level comparisons against a single reference image. It uses explicit required conditions and protected conditions to define what must change and what must remain intact:

• Required conditions: Has the specified editing goal been correctly achieved?

• Protected conditions: Have the elements, background, and spatial structures that should remain unchanged been preserved?

The team evaluated nine image-to-image models and two image-to-video models, including GPT-Image-2, Seedream 5.0 Pro, and Gemini 3.1 Flash Image. Video outputs were evaluated using their final frames. Among the 11 systems, GPT-Image-2 achieved the highest SolveScore at 57.0%. The results reveal substantial errors in both task completion and content preservation, even for the strongest evaluated model. They also show that visual quality alone does not establish whether a problem has been solved correctly. 2.2 Think Before You Edit: SolveEdit-Plan To address this bottleneck, the team introduced an explicit visual planning stage before generation—SolveEdit-Plan—rather than retraining the underlying foundation model:

• Step 1: Identify information left unspecified in the instruction and the underlying goals.

• Step 2: Gather visual evidence from the scene and compare possible editing approaches through reasoning.

• Step 3: Translate the selected solution into concrete editing actions and descriptions of the regions to preserve, then pass the resulting instruction to the generator for a single generation call.

Without any changes to the underlying generator, adding this planning stage increased GPT-Image-2’s SolveScore from 57.0% to 71.6%, a gain of 14.6 percentage points. This improvement demonstrates the value of reasoning under explicit constraints in reducing unintended changes during image editing.

3. What Comes Next: More Reliable Visual Problem Solving

SolveEdit broadens the evaluation of generative models beyond visual appearance. It asks whether a model can understand the current scene, infer a valid transformation, and produce the intended result while preserving unrelated content.

For creators, progress on these capabilities could mean fewer repeated attempts and more reliable editing. A useful visual assistant needs to understand the goal, identify the relevant evidence, determine the scope of the change, and execute it accurately.

The results demonstrate a promising role for planning, while leaving substantial room for improvement. Producing an attractive image remains valuable; solving the underlying visual problem is an additional requirement for dependable creative tools.

Paper and code are publicly available. According to the repository’s release notes, benchmark images and atomic contracts will be distributed separately. Researchers and developers are welcome to explore the work and contribute to further research on visual problem solving.

ZODA also welcomes research collaborations with universities, companies, and research institutes to co-develop high-quality open datasets and benchmarks in areas including world models, AI agents, and embodied intelligence. Get in touch to explore opportunities for collaboration.