Are Deep Learning Models Pre-trained on RGB Data Good Enough for RGB-Thermal Image Retrieval?

Amulya Pendota, Sumohana S. Channappayya; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 4287-4296

RGB-Thermal (RGB-T) image retrieval is crucial in scenarios where RGB data alone is insufficient for reliable decision-making. These include all-day all-weather surveillance and security operations search and rescue operations and autonomous navigation systems. However RGB-T image retrieval remains underexplored due to the nature of the currently available datasets. Specifically these datasets do not lend themselves to training models in the standard RGB visual place recognition (VPR) setting. Therefore we explore and analyse the effectiveness of existing RGB pre-trained models in addressing the RGB-T image retrieval problem. In particular we evaluate the performance of numerous pre-trained models on the RGB-T image retrieval task. The efficacy of the models is evaluated on eight RGB-T datasets. Quantitatively recall rates Central Kernel Alignment (CKA) and the proposed centroid condition are used for evaluation. Qualitative analysis uses distance plots t-SNE plots and heatmaps like Saliency Based Similarity Maps (SBSM). Interestingly and surprisingly some of the pre-trained models deliver good cross-domain retrieval performance. To the best of our knowledge this analysis is the first of its kind in RGB-T image retrieval with the available RGB-T datasets. We believe this will serve as a baseline for future work in this area of research.