DECNet: A Non-Contacting Dual-Modality Emotion Classification Network for Driver Health Monitoring

Zhekang Dong, Chenhao Hu, Shiqi Zhou, Liyan Zhu, Junfan Wang, Yi Chen, Xudong Lv, Xiaoyue Ji; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 371-379

Negative emotions have been identified as significant factors influencing driver behavior, easily leading to extremely serious traffic accidents. Hence, there is a pressing need to develop an automatic emotion classification method for driver health monitoring and road safety improvement. Most of the existing methods predominantly focus on single modalities, resulting in suboptimal classification performance due to the underutilization of heterogeneous information. In this work, we propose a novel non-contacting dual-modality driver emotion classification network (DECNet) to address these limitations. DECNet consists of three key modules: 1) facial video modality processing module; 2) driving behavior modality processing module; 3) fusion decision module. Meanwhile, we introduce a combined multi-task learning strategy within DECNet to improve the efficacy in the driver emotion classification task. To evaluate the effectiveness of the proposed DECNet, we conducted experiments on the PPB-Emo dataset, the experimental results showcase the superiority in terms of accuracy (>= 6.12% Acc-7) and F1-score (>= 7.25% F1-7) compared to existing state-of-the art methods. The model and code will be available at https://github.com/fqfqngxhs/DECNet.git