SafeSO: Interpretable and Explainable Deep Learning Approach for Seat Occupancy Classification in Vehicle Interior

Joanna Jaworek-Korjakowska, Aleksander Kostuch, Pawel Skruch; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021, pp. 103-112

Classification of seat occupancy in in-vehicle interior remains a significant challenge and is a promising area in the functionality of new generation cars. As majority of accidents are related to the driver errors the consequences of not wearing, or improperly wearing, a seat belt are clear. The NHTSA reports that 47% of the 22,215 passenger vehicle occupants killed in 2019 were not wearing seat belts. To address this problem we propose a deep learning based framework to classify seat occupancy into seven most important categories. In this study, we present an interpretable and explainable AI approach that takes advantage of pre-trained networks including ResNet152V2, DenseNet121 and the most recent EfficientNetB0-B5-B7 to calculate the feature vectors followed by an adjusted densely-connected classifier. Our model provides an interpretation of its results through the identification of object parts without direct supervision and their contribution towards classification. We explore and propose two new statistical metrics including HGD_ score and HGDA_score which are based on the multivariate Gaussian distribution for assessing heatmaps without using human-annotated object parts to quantify the interpretability of our network. We demonstrate that the calculated statistical metrics lead to an interpretable model that correlates with the framework accuracy and can flexibly analyze heatmaps at any resolution for different user needs. Furthermore, extensive experiments have been performed on the SVIRO database including 7,500 sceneries for BMW X5 model which confirm the ability of the developed framework based on the EfficientNetB5 architecture to classify seat occupancy into seven main categories with 79.87% overall accuracy as well as 95.92% recall and 90.32% specificity for empty seats recognition, which is a state-of-the-art result in this domain.