ART-V: Auto-Regressive Text-to-Video Generation with Diffusion Models

Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, Chong Luo, Yueyi Zhang, Zhiwei Xiong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 7395-7405

We present ART-V an efficient framework for auto-regressive video generation with diffusion models. Unlike existing methods that generate entire videos in one-shot ART-V generates a single frame at a time conditioned on the previous ones. The framework offers three distinct advantages. First it only learns simple continual motions between adjacent frames therefore avoiding modeling complex long-range motions that require huge training data. Second it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications. Third it can generate arbitrarily long videos conditioned on a variety of prompts such as text image or their combinations making it highly versatile and flexible. To combat the common drifting issue in AR models we propose masked diffusion model which implicitly learns which information can be drawn from reference images rather than network predictions in order to reduce the risk of generating inconsistent appearances that cause drifting. Moreover we further enhance generation coherence by conditioning it on the initial frame which typically contains minimal noise. This is particularly useful for long video generation. When trained for only two weeks on four GPUs ART-V already can generate videos with natural motions rich details and a high level of aesthetic quality. Besides it enables various appealing applications e.g. composing a long video from multiple text prompts.