Complete and Temporally Consistent Video Outpainting

Loïc Dehan, Wiebe Van Ranst, Patrick Vandewalle, Toon Goedemé; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022, pp. 687-695

We describe a novel method for video outpainting. The goal of outpainting is to fill in missing regions at the edges of video frames. Our focus lies on converting portrait (9:16) to landscape (16:9) video. In contrast, most video completion research is focused on inpainting: filling a masked section within the frame based on the remaining, known pixels. Our proposed method consists of three main aspects: (1) We form a background estimation using video object segmentation and video inpainting methods, (2) we use optical flow to form temporal consistency, and (3) we propose image shifting to improve individual frame completions. Our method is able to successfully broaden the aspect ratio of a video. On most videos, we achieve realistic results. Only on videos with complex camera motion and foreground objects leaving the frame, the quality is less. In contrast to other state-of-the-art methods, our method is able to reconstruct the full frame, including unseen image parts. Moreover, it is temporally consistent. We evaluate our method on the DAVIS and YouTube-VOS datasets. The code is publicly available.