Skin Lesion Classification Using Dermoscopic Images and Clinical Metadata: Insights from Multimodal Models

Sakib Ahammed, Xia Cui, Wenqi Lu, Moi Hoon Yap; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2025, pp. 222-230

State-of-the-art methods primarily use dermoscopic skin images to classify lesions. Incorporating clinical metadata to improve melanoma diagnosis has increased popularity in the community. This study explores whether a single modality (dermoscopic images) or multimodality (dermoscopic images and clinical metadata) enhances performance for skin lesion classification. In particular, it tries to answer the question by comparing three multimodal architectures. This paper introduces the ISIC-DICM-17K dataset, a curated balanced dataset comprising 17,060 dermoscopic images and clinical metadata with an equal distribution of melanoma and non-melanoma classes. We evaluate the impact of including clinical metadata on multimodal model performance using both supervised and transfer learning models. Our results show that including metadata significantly enhances supervised learning models' performance, while all transfer learning models outperform supervised learning models with 3% to 12% performance improvement. Our statistical and visual analyses highlight the importance of clinical metadata in improving feature clustering and class separability, and the HeatMap_index analysis shows the effectiveness in identifying relevant features on lesion images. Our GitHub repository (accessible at https://github.com/mmu-dermatology-research/isic-dicm-17k) contains the ISIC-DICM-17K dataset and image IDs referencing the original ISIC dataset sources.