Unsupervised Model-Based Learning for Simultaneous Video Deflickering and Deblotching

Anuj Fulari, Satish Mulleti, Ajit Rajwade; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 4117-4125

Vintage videos, as well as modern day videos acquired at high frame rates, suffer from a visually disturbing artifact called flicker, which is the rapid change in average intensity across consecutive frames. Vintage videos also suffer from blotch artifacts, i.e., each video frame contains small regions at random locations with undefined pixel values. We present a model-based learning approach to remove flicker as well as blotches simultaneously. Our work uses a pixel-wise affine intensity model for flicker between neighboring frames, with coefficients that vary smoothly in the spatial sense but randomly across time. Due to smooth spatial variation, the flicker coefficients for any given frame can be modelled as linear combinations of low frequency discrete cosine transform (DCT) bases. We also model blotches as heavy-tailed but sparse artifacts affecting every frame. We then present a novel framework to restore the video frames by jointly estimating the blotches as well as the DCT coefficients of the flicker, via convex optimization. Given the high computational cost of the optimization based method for processing an entire video, we use a deep unrolled neural network approach to achieve similar restoration quality at significantly reduced cost. Our approach is completely unsupervised and model based, and hence simple and interpretable. It produces high quality reconstructions, in terms of visual appeal as well as numerical metrics, on a variety of vintage videos as well as high speed videos. It does not suffer from generalization issues unlike some recent state of the art supervised methods which use end to end neural networks for restoration.