RealPixVSR: Pixel-Level Visual Representation Informed Super-Resolution of Real-World Videos

Tony Nokap Park, Yunho Jeon, Taeyoung Na; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2024, pp. 421-430

Recently, there have been significant advances in video super-resolution (VSR) techniques under blind and practical degradation settings. These techniques restore the fine details of each video frame while maintaining the temporal consistency between frames for a smooth motion. Unfortunately, many attempts still fall short in the case of real-world videos. When diverse and complex in-the-wild degradation is introduced, the task becomes non-trivial and challenging. As a result, VSR techniques perform poorly in general. We argue that there is more space to improve the performance of VSR methods, as current methods are only trained on image-level degradation settings, leading to a restoration quality that may be sub-optimal for real-world degradation that varies pixel-wise within an image. To this end, we propose RealPixVSR which leverages the pixel-level representations to improve the pixel-level sensitivity to degradation. The pixel-level content-invariant degradation representation is learned in a self-supervised manner using the contrastive learning network referred to as the Pixel-Degradation-Representation-Network (PDRN). And the learned visual representation is merged with the cleaning and restoration networks using the Pixel-Degradation-Informed-Block (PDIB). Through experiments, we show that our network outperforms the latest state-of-the-art VSR models for real-world video