Multi-Modal Information Fusion for Action Unit Detection in the Wild

Yuanyuan Deng, Xiaolong Liu, Liyu Meng, Wenqiang Jiang, Youqiang Dong, Chuanhe Liu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 5855-5862

Action Unit (AU) detection is an important research branch in affective computing, which better understands human emotional intentions and responds more naturally to their needs and desires. In this paper, we present our latest progress techniques in the 5th Affective Behavior Analysis in-the-wild (ABAW) competition, including data balancing by marking, extracting features visual through models trained in face database and audio through deep networks and traditional methods, proposing model structures for mapping multimodal information to a unify multimodal vector space and fusing results from multiple models. These methods are effective on the official validation dataset of the Aff-Wild2. The final F1 in the 5th ABAW competition test dataset achieves 54.22%, 4.33% higher than the best results in the 3rd ABAW competition.