## TL;DR

We propose Through-The-Mask, a two-stage framework for Image-to-Video generation that uses mask-based motion trajectories to enhance object-specific motion accuracy and consistency, achieving state-of-the-art results, particularly in multi-object scenarios.

## Videos (DiT-based Model)

The metallic robot with blue eyes performs a series of push-ups.
The bear strolls through the forest, taking slow, deliberate steps.
A monster and a squirrel shake hands gently, and continue moving while shaking hands.
The left and right meerkats play together.
The gibbon sits on a rock, raising a hand as if waving to an audience.
The gorilla strums the guitar rhythmically.
One parrot gracefully stretches its wings while the other looks on.
The gorilla raises both arms high above its head, stretching with a serene expression.

## Videos (U-Net-based Model)

The race car on the track speeds up.
Two vibrant parrots bob their heads in unison.
The ocean waves gently rise and fall.
The bug car drives forward.
One shark glides smoothly near the seafloor, while the other shark swims upward.
A snowman in a forest caught in a burst of flames.
A serene waterfall cascades gently over rocks, surrounded by lush greenery.
The fox gracefully strolls across the snowy landscape.

## Abstract

We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description. While recent advancements produce photorealistic outputs, they frequently struggle to create videos with accurate and consistent object motion, especially in multi-object scenarios. To address these limitations, we propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation. Our key innovation is the introduction of a mask-based motion trajectory as an intermediate representation, that captures both semantic object information and motion, enabling an expressive but compact representation of motion and semantics. To incorporate the learned representation in the second stage, we utilize object-level attention objectives. Specifically, we consider a spatial, per-object, masked-cross attention objective, integrating object-specific prompts into corresponding latent space regions and a masked spatio-temporal self-attention objective, ensuring frame-to-frame consistency for each object. We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art results in temporal coherence, motion realism, and text-prompt faithfulness. Additionally, we introduce a new challenging benchmark for single-object and multi-object I2V generation and demonstrate our method's superiority on this benchmark.

## Method

Overview of our I2V framework, transforming a reference image \( x^{(0)} \) and text prompt \( c \) into a coherent video sequence \( \hat{x} \). A pre-trained LLM is used to derive the motion-specific prompt \( c_{motion} \) and object-specific prompts \( c_{local} = \{c_{local}^{(1)}, \dots, c_{local}^{(L)}\} \) capturing each object's intended motion. We generate an initial segmentation mask \( s^{(0)} \) from \( x^{(0)} \) using SAM2. In Stage 1, the Image-to-Motion utilizes \( x^{(0)}, s^{(0)}, c_{motion} \) to generate mask-based motion trajectories \( \hat{s} \) that represent object-specific movement paths. In Stage 2, the Motion-to-Video takes as input \( x^{(0)}, \hat{s}, c \) as a global condition, and \( c_{local} \) through masked attention blocks, integrating the mask-based motion trajectory softly using learned attention layers. Specifically, we employ (i) masked cross-attention to integrate object-specific prompts into corresponding latent space regions, and (ii) masked self-attention to ensure each object maintains consistency across frames by restricting attention to object-defined regions.

## Qualitative Comparisons: DiT-based Model

The aliens dance the robot dance.
Input image
Ours
TI2V
The car races up the mountain road.
Input image
Ours
TI2V
The orangutan extends both arms and performs a peaceful pose.
Input image
Ours
TI2V
The red car on the right speeds away, leaving a cloud of dust behind, while the black car on the left moves forward slowly.
Input image
Ours
TI2V
One snake threatens the other, while the other snake rests.
Input image
Ours
TI2V

## Qualitative Comparisons: U-Net-based Approaches

Two emus jog side by side through a forest path.
Input image
Ours
TI2V
ConsistI2V
Motion-I2V
DynamiCrafter
VideoCrafter
The gorilla strums the guitar rhythmically.
Input image
Ours
TI2V
ConsistI2V
Motion-I2V
DynamiCrafter
VideoCrafter
A happy penguin rolling along on bright pink roller skates, wings flapping to keep steady as it moves.
Input image
Ours
TI2V
ConsistI2V
Motion-I2V
DynamiCrafter
VideoCrafter
The robot swims forward smoothly.
Input image
Ours
TI2V
ConsistI2V
Motion-I2V
DynamiCrafter
VideoCrafter
The two kangaroos stand up in perfect synchronization.
Input image
Ours
TI2V
ConsistI2V
Motion-I2V
DynamiCrafter
VideoCrafter

## Mask vs Flow-based Representations

We compare our method using a mask-based trajectory to our method using flow-based trajectory.

The chimpanzees hold each other's hands and move their free hands up and down.
Input image
Generated mask
Mask-based model (Ours)
Generated flow
Flow-based model
The jeep rumbles forward on the winding mountain road.
Input image
Generated mask
Mask-based model (Ours)
Generated flow
Flow-based model

## BibTex

    @misc{yariv2025throughthemaskmaskbasedmotiontrajectories,
          title={Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation},
          author={Guy Yariv and Yuval Kirstain and Amit Zohar and Shelly Sheynin and Yaniv Taigman and Yossi Adi and Sagie Benaim and Adam Polyak},
          year={2025},
          eprint={2501.03059},
          archivePrefix={arXiv},
          primaryClass={cs.CV},
          url={https://arxiv.org/abs/2501.03059},
    }