A Dual Branch Network for Emotional Reaction Intensity Estimation

Jun Yu, Jichao Zhu, Wangyuan Zhu, Zhongpeng Cai, Guochen Xie, Renda Li, Gongpeng Zhao, Qiang Ling, Lei Wang, Cong Wang, Luyu Qiu, Wei Zheng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 5811-5818

Abstract

Emotional Reaction Intensity(ERI) estimation is an important task in multimodal scenarios, and has fundamental applications in medicine, safe driving and other fields. In this paper, we propose a solution to the ERI challenge of the fifth Affective Behavior Analysis in-the-wild(ABAW), a dual-branch based multi-output regression model. The spatial attention mechanism is used to better extract visual features, and the Mel-Frequency Cepstral Coefficients technology extracts acoustic features. Temporal Encoder is composed of Temporal Convolutional Network and Transformer Encoder, which is used to capture the temporal relationship between features. And a method named modality dropout is added to fusion multimodal features. Our approach for ERI challenge achieves Pearson's Correlation Coefficient of 0.4439 on the validation set and 0.4380 on the test set, which ranks second in the final leaderboard.

Related Material

[pdf] [arXiv]
[bibtex]
@InProceedings{Yu_2023_CVPR, author = {Yu, Jun and Zhu, Jichao and Zhu, Wangyuan and Cai, Zhongpeng and Xie, Guochen and Li, Renda and Zhao, Gongpeng and Ling, Qiang and Wang, Lei and Wang, Cong and Qiu, Luyu and Zheng, Wei}, title = {A Dual Branch Network for Emotional Reaction Intensity Estimation}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, month = {June}, year = {2023}, pages = {5811-5818} }