Multimodal Fusion Learning with Dual Attention for Medical Imaging

Joy Dhar, Nayyar Zaidi, Maryam Haghighat, Sudipta Roy, Puneet Goyal, Azadeh Alavi, Vikas Kumar; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 4362-4371

Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However existing methods face three key limitations. First they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally they typically rely on a single attention mechanism missing the benefits of multiple attention strategies within and across various modalities. To address these issues this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules - i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network forming a multi modal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality such as dermoscopy pap smear MRI and CT-scan whereas multimodal information fusion attention module learns more refined multimodal shared representations - improving the network's generalization across multiple tasks and enhancing overall performance. Additionally to estimate the uncertainty of DRIFA-Net predictions we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.