Does the Fairness of Your Pre-Training Hold Up? Examining the Influence of Pre-Training Techniques on Skin Tone Bias in Skin Lesion Classification

Pratinav Seth, Abhilash K. Pai; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2024, pp. 570-577

Deep Neural Networks (DNNs) have found widespread application in various domains, but the challenge of addressing Algorithmic bias and ensuring fairness in their decision-making processes has emerged as a critical concern, particularly in mission-critical contexts. One of the main reasons for this concern is the inadequate representation of certain groups in the available datasets used for training. Pre-Training is a powerful technique for training DNNs, but it can be affected by pre-existing biases in the dataset. These biases can be transferred to the DNN during Pre-Training, leading to the DNNs making biased decisions, even when trained on unbiased datasets. This study investigates the impact on the fairness of popular Pre-Training methods, such as Masked Image Modeling (MAE, SimMIM) and Self-Supervised Learning (BYOL, MoCo, SimCLR, VICRegL), when used on skin lesion classification datasets with underrepresented demographic groups. The study compares the performance of pre-trained models to supervised learning backbones on two skin lesion datasets (ISIC-2019 and Fitzpatrick17k) with different skin tone distributions. The findings of this study reveal that Pre-Training improves performance but has a trade-off with fairness, which can be a potential danger associated with the model when applied in the real world. This study is one of the first to investigate how Self-Supervised Learning and Masked Image Modeling Pre-Training methods affect fairness in both in-distribution and out-of-distribution scenarios.