SwinFuSR: An Image Fusion-inspired Model for RGB-guided Thermal Image Super-resolution

Cyprien Arnold, Philippe Jouvet, Lama Seoud; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 3027-3036

Thermal imaging plays a crucial role in various applications but the inherent low resolution of commonly available infrared (IR) cameras limits its effectiveness. Conventional super-resolution (SR) methods often struggle with thermal images due to their lack of high-frequency details. Guided SR leverages information from a high-resolution image typically in the visible spectrum to enhance the reconstruction of a high-res IR image from the low-res input. Inspired by SwinFusion we propose SwinFuSR a guided SR architecture based on Swin transformers. In real world scenarios however the guiding modality (e.g. RBG image) may be missing so we propose a training method that improves the robustness of the model in this case. Our method has few parameters and outperforms state of the art models in terms of Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM). In Track 2 of the PBVS 2024 Thermal Image Super-Resolution Challenge it achieves 3rd place in the PSNR metric. Our code and pretained weights are available at https://github.com/VisionICLab/SwinFuSR.