Towards Calibrated Multi-label Deep Neural Networks

Jiacheng Cheng, Nuno Vasconcelos; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27589-27599

The problem of calibrating deep neural networks (DNNs) for multi-label learning is considered. It is well-known that DNNs trained by cross-entropy for single-label or one-hot classification are poorly calibrated. Many calibration techniques have been proposed to address the problem. However little attention has been paid to the calibration of multi-label DNNs. In this literature the focus has been on improving labeling accuracy in the face of severe dataset unbalance. This is addressed by the introduction of asymmetric losses which have became very popular. However these losses do not induce well calibrated classifiers. In this work we first provide a theoretical explanation for this poor calibration performance by showing that these loses losses lack the strictly proper property a necessary condition for accurate probability estimation. To overcome this problem we propose a new Strictly Proper Asymmetric (SPA) loss. This is complemented by a Label Pair Regularizer (LPR) that increases the number of calibration constraints introduced per training example. The effectiveness of both contributions is validated by extensive experiments on various multi-label datasets. The resulting training method is shown to significantly decrease the calibration error while maintaining state-of-the-art accuracy.