Towards Robust Video Frame Interpolation with Long-Term Propagation

Ziqi Huang, Kelvin C.K. Chan, Bihan Wen, Ziwei Liu; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2024, pp. 621-636

Video frame interpolation aims to synthesize non-existent frames between two consecutive frames in a video, and its importance can be seen from its wide applications in computer vision. The key to video frame interpolation is predicting the intermediate motions between two given frames, so that the synthesized frames are coherent with the input video. However, the existing approaches of imposing assumptions on motions, such as linear and quadratic trajectories, are not generalizable to complex motions in real-world videos. In this work, we propose long-term propagation for robust and general motion prediction in video frame interpolation. To more thoroughly understand the motion trajectories, we propose to implicitly track the motion paths through a long sequence of video frames. The motion features are then used to refine the motion predicted from the primitive motion assumptions. Our proposed long-term propagation of motion can be easily integrated into existing video frame interpolation approaches. Quantitative and qualitative results demonstrate that long-term propagation effectively improves interpolation performance when incorporated into various state-of-the-art baselines. In addition, we present a series of analytical experiments to study the mechanism, advantages, and limitations of long-term propagation in video frame interpolation to inspire future works.