Spectral Modality-Aware Interactive Fusion Network for HSI Super-Resolution

Meng Xu, Jiayou Mao, Ziqian Mo, Xiyou Fu, Sen Jia; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 4544-4560

Due to the limitations of current spectral imaging equipment in acquiring high-resolution hyperspectral images (HR-HSIs), a common approach is to fuse low-resolution hyperspectral images (LR-HSIs) with high-resolution multispectral images (HR-MSIs). However, most existing methods have not fully taken into account the correlation and discrepancy in modality between HSIs and MSIs. To address this limitation, we propose an innovative spectral modality-aware interactive fusion network (SMIF-NET) for comprehensive extraction of spectral information and seamless feature fusion. First, we introduce the spectral modality-aware transformer (SMAT) with a dual-attention mechanism to compute spectral self-similarity and cross-spectral correlation. Second, we apply the interactive spatial-spectral feature fusion (IS2F2) to fuse the acquired high-level spectral and spatial features. This fusion technique combines spatial-wise and channel-wise squeeze and excitation to achieve seamless integration of spatial-spectral information. Finally, the extensive experiments on three datasets demonstrate the superior performance of SMIF-NET in both visual and quantitative assessments compared to eight state-of-the-art (SOTA) fusion-based methods.