AlignVid: Taming Visual Dominance via Training-Free Attention Modulation in Text-guided Image-to-Video Generation

When we give a video generation model an image of a house and enter the prompt, “Make a sunflower grow in front of the house,” it may seem, intuitively, like a simple video editing task. But in today’s text-guided image-to-video tasks, this kind of request is not always executed accurately. Models often preserve the reference image well, but ignore the key change requested in the text. The house remains the same house, and the scene looks stable, but the instruction has not truly been completed.

This is the question addressed by AlignVid: Taming Visual Dominance via Training-Free Attention Modulation in Text-Guided Image-to-Video Generation, a paper with participation from ZODA. The work was jointly completed by researchers from the Hong Kong University of Science and Technology, the ZODA International Open Data Community, the University of Central Florida, the Beijing Academy of Artificial Intelligence, the Chinese University of Hong Kong, the Hong Kong University of Science and Technology (Guangzhou), and other institutions. It has been accepted to ICML 2026.

As a community focused on data, evaluation, and open-source collaboration, ZODA cares not only about high-quality datasets themselves, but also about how data can be used to evaluate model capabilities more accurately, reveal model limitations, and further drive improvements in alignment, controllability, and inference efficiency for generative models. AlignVid was developed in this context. It focuses on the problem of “semantic negligence” in text-guided image-to-video generation, proposes a training-free inference-time attention modulation method, and introduces the OmitI2V benchmark to evaluate whether models truly execute the visual edits specified by text.

1. Strong Image Priors: The Problem of Semantic Negligence in Image-to-Video Generation

In image-to-video generation, the reference image contains a large amount of high-fidelity visual information. It helps the model preserve the subject, background, and overall visual consistency, but it can also introduce a side effect: when the image information is too strong, it dominates the attention mechanism, making it difficult for the text instruction to truly influence the generated result.

We refer to this phenomenon as visual dominance. In the generated output, it appears as semantic negligence: the generated video follows the original image more than the requested additions, deletions, or modifications in the text.

This is not an isolated issue in a single model. We evaluated several current image-to-video models on the OmitI2V benchmark, and the results show that no model maintained stable performance across all three editing tasks: modification, addition, and deletion. For example, Wan2.1 performs relatively well on modification and addition tasks, but still drops significantly on deletion tasks. FramePack and its variants also show limitations in semantic alignment. This suggests that, in text-guided image-to-video tasks, whether models can accurately understand and execute fine-grained semantic instructions remains a problem that deserves systematic study.

2. A Counterintuitive Clue: Slight Blur Can Help

We first conducted a preliminary experiment. We found that after applying a slight Gaussian blur to the input image, the model’s semantic following and motion degree actually improved.

This does not mean that “blurring the image” is itself a good solution. Blurred input damages the quality and details of the reference image. What is truly valuable is the mechanism behind this phenomenon: blur weakens the dominance of high-fidelity image tokens, allowing the model’s attention to be redistributed and giving text tokens more opportunity to influence the generation process.

Across 30 samples, we found that after blurring, the attention scores of the image, video, and text modalities all increased, while attention entropy decreased. In other words, attention became more concentrated, and the text signal was less easily suppressed by the image prior.

This made the research question clearer: can we avoid damaging the input image and instead directly adjust the attention distribution inside the model, simulating the effect of “reducing entropy and releasing the text signal”?

3. AlignVid: Adjusting Attention at Inference Time Without Training a New Model

The core idea of AlignVid is to avoid retraining the model, avoid introducing external masks, and avoid modifying the input image. Instead, it modulates the model’s internal attention during inference.

The method consists of two components.

The first is Attention Scaling Modulation, or ASM. It changes the effective temperature of the softmax function by scaling the query or key vector embeddings in the attention mechanism, making the attention distribution more concentrated. From the perspective of energy-based models, we view attention as an energy distribution based on the softmax function and show that increasing the scaling coefficient reduces conditional entropy. Lower entropy means attention is no longer overly dispersed, giving text editing tokens a greater chance to compete with dense visual priors.

The second is Guidance Scheduling, or GS. If strong intervention is applied to all Transformer modules and all denoising steps, semantic constraints become stronger, but visual quality may also be affected. Therefore, AlignVid selectively applies attention scaling modulation at more critical positions: module-level guidance scheduling selects modules that are more sensitive to the foreground, while step-level guidance scheduling emphasizes earlier denoising steps, because early stages more strongly influence the overall semantic layout.

As a result, AlignVid is more like a lightweight, plug-and-play inference-time adjustment method than a new model that requires retraining. It does not require model retraining and only takes effect during inference. It does not require additional masks, nor does it damage the quality or details of the input image. The core of the method is to adjust the attention distribution through a scaling coefficient. In the experiments, this coefficient is set to γ = 1.35 to achieve a better balance between semantic following and visual quality.

4. OmitI2V: Building an Evaluation Set for “Whether the Edit Was Actually Executed”

To evaluate whether text-guided image-to-video generation truly completes the instruction, we introduce the OmitI2V benchmark.

This benchmark contains 367 image-text pairs, covering three clearly defined visual editing tasks: modification, addition, and deletion. The data includes different types of images, such as real images, synthetic images, and animated images.

For evaluation, we use a visual-question-answering-based yes/no judgment mechanism to determine whether the generated video completes the edit required by the text. For example, when the text instruction asks for “a sunflower to appear in front of the house,” the evaluation question checks whether a sunflower really appears in front of the house. In addition, we evaluate the motion degree and aesthetic quality of the video to measure whether the generated video has sufficient dynamic change and whether the overall visual effect is natural and aesthetically pleasing.

This point is critical. Traditional video generation evaluation often focuses more on overall quality, visual stability, or coarse-grained text alignment. But for fine-grained editing tasks such as “whether a specific object was actually added,” “whether an element was actually removed,” or “whether the specified content was modified as required,” traditional metrics do not provide sufficient signal.

The value of OmitI2V lies in turning the problem of “semantic negligence” in image-to-video generation into an evaluation problem that can be systematically observed, quantified, and compared.

Figure 4. Proportions of the three task types: modification, addition, and deletion

5. Why This Is Also a Problem ZODA Cares About

For ZODA, the value of this work is not only that it proposes a new inference-time adjustment method, but also that it turns a previously hard-to-measure problem into a task that can be evaluated, compared, and reproduced.

In many evaluations of generative models, it is often easier to observe and score whether a video is clear, whether the frames are coherent, and whether the overall result is aesthetically pleasing. But for text-guided image-to-video tasks, another equally important question is: did the model actually execute the editing intent in the text?

For example, did it really add a new object, remove a specific element, or change an object’s attributes?

The OmitI2V benchmark is built around this question. It focuses not only on the visual quality of generated videos, but also uses explicit edit types and a yes/no judgment mechanism to observe whether the model completes the semantic change specified by the text prompt.

This is highly aligned with the direction ZODA is advancing: using more fine-grained and more interpretable evaluation tasks to help researchers and developers see the real boundaries of model capabilities. Progress in generative models should not be defined only by “whether the image looks better,” but also by “whether the model can reliably execute instructions.”

6. Results: Consistent Semantic Improvements Across Multiple Models

In the experiments, we inserted AlignVid into the inference pipelines of FramePack, FramePack F1, and Wan2.1. The results show that AlignVid improves performance across all three semantic editing tasks.

On FramePack, the modification score increased from 64.99 to 68.22, the addition score increased from 68.55 to 73.13, and the deletion score increased from 58.14 to 60.21. Motion degree increased from 20.05 to 28.53, while aesthetic quality slightly decreased from 63.94 to 63.57.

On FramePack F1, the modification score increased from 64.45 to 71.27, the addition score increased from 67.79 to 71.60, and the deletion score increased from 58.50 to 61.06. Motion degree increased from 24.42 to 33.16, while aesthetic quality decreased from 63.10 to 62.10.

On Wan2.1, the modification score increased from 72.35 to 77.20, the addition score increased from 71.75 to 79.54, and the deletion score increased from 63.13 to 69.47. Motion degree increased from 46.02 to 47.04, while aesthetic quality decreased from 63.12 to 61.63.

These results show that AlignVid is not effective only on a single model or a single editing type. Instead, it brings relatively consistent semantic alignment improvements across multiple base models and multiple editing tasks. It is important to note that while the method improves text instruction following, aesthetic quality shows a slight decline in some settings. Therefore, AlignVid’s core contribution is not that it “solves all problems at no cost,” but that it provides a controllable inference-time trade-off among semantic following, motion generation, and visual quality.

7. Minimal Additional Overhead

For an inference-time method, computational overhead is also important.

AlignVid introduces runtime overhead of only +0.09%, +0.03%, and +0.01% on FramePack, FramePack F1, and Wan2.1, respectively. This means it adds almost no extra computational cost.

This is one reason AlignVid is worth paying attention to: it does not improve results by retraining a larger model, but by redistributing attention inside existing models, allowing text editing signals to occupy a more appropriate position during generation.

8. Conclusion: From Generated Results to the Generation Process

The significance of AlignVid is not only that it makes certain video examples better match the prompt. More importantly, it reveals a mechanistic issue in text-guided image-to-video generation: when the image prior is too strong, the model may consistently produce videos that look reasonable but do not execute the instruction.

This raises new requirements for generative model evaluation. In the future, evaluating image-to-video models should not only consider whether the images are clear and whether the motion is natural, but also whether the model truly follows the editing intent specified in the text.

From ZODA’s perspective, AlignVid and OmitI2V together provide a clear example: high-quality data, fine-grained evaluation, and model mechanism analysis are not separate efforts. OmitI2V turns “semantic negligence” into an evaluable task, while AlignVid further explores how visual dominance can be mitigated from the perspective of attention mechanisms without retraining the model.

This is also a direction ZODA continues to focus on: helping the community understand model capabilities more accurately, and see more clearly the problems that models have not yet solved, through open data, evaluation benchmarks, and research outcomes.

When we ask a model to “make a sunflower grow in front of the house,” what truly matters is not only whether the video looks smooth, but whether the sunflower actually appears.

Follow ZODA for more updates on AI data, model evaluation, and the open research ecosystem. We also look forward to working with researchers, developers, and industry partners to build more reliable, interpretable, and reproducible AI evaluation infrastructure.