Skin tone Diagnosis in the Wild: Towards More Robust and Inclusive User Experience Using Oriented Aleatoric Uncertainty

Emmanuel Malherbe, Michel Remise, Shuai Zhang, Matthieu Perrot; Proceedings of the Asian Conference on Computer Vision (ACCV), 2022, pp. 1-19

The past decade has seen major advances in deep learning models that are trained to predict a supervised label. However, estimating the uncertainty for a predicted value might provide great information beyond the prediction itself. To address this goal, using a probabilistic loss was proven efficient for aleatoric uncertainty, which aims at capturing noise originating from the observations. For multidimensional predictions, this estimated noise is generally a multivariate normal variable, characterized by a mean value and covariance matrix. While most of literature have focused on isotropic uncertainty, with diagonal covariance matrix, estimating full covariance brings additional information, such as the noise orientation in the output space. We propose in this paper a specific decomposition of the covariance matrix that can be efficiently estimated by the neural network. From our experimental comparison to the existing approaches, our model offers the best trade-off between uncertainty orientation likeliness, model accuracy and computation costs. Our industrial application is skin color estimation based on a selfie picture, which is at the core of an online make-up assistant but is a sensitive topic due to ethics and fairness considerations. Thanks to oriented uncertainty, we can reduce this risk by detecting uncertain cases and proposing a simplified color correction bar, thus making user experience more robust and inclusive.