OmniFusion: Exemplar-based Video Colorization Using OmniMotion and Diffusion Priors

Xiaoyuan Fang, Longquan Dai, Jinhui Tang; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 1215-1232

Exemplar-based video colorization is a challenging task that involves the consistent propagation of colors across frames and the reasonable inference of colors from grayscale within frames. This paper proposes a novel video colorization method called OmniFusion, which iteratively completes the video colorization through two following steps. In the inter-frame propagation step, OmniMotion establishes correspondences between pixels across video frames. Any grayscale pixel can be queried whether a corresponding pixel and color are available from the exemplar according to such correspondences. Consequently, the processed images may still contain regions lacking color. In the intra-frame inpainting step, diffusion model provides these grayscale regions in a frame with plausible colors. The colorized frame is then fed into the first step as an exemplar, accepting queries from all uncolored pixels. This iterative process continues until all pixels are colorized. Evaluations indicate that OmniFusion achieves excellent performance in video colorization, surpassing existing methods in terms of color fidelity and visual quality.