GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models

Sai Sree Harsha, Ambareesh Revanur, Dhwanit Agarwal, Shradha Agrawal; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 7559-7568

Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes where existing approaches fail.