Uncovering Hidden Emotions with Adaptive Multi-Attention Graph Networks

Ankith Jain Rakesh Kumar, Bir Bhanu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 4822-4831

Micro-expressions (MEs) are subtle expressions lasting a fraction of a second offering valuable cues for understanding human emotions and intentions. However effectively classifying these subtle expressions from video data poses several challenges due to their short duration and low intensity. This paper addresses these issues and presents a novel 2-stream Adaptive Multi-Attention ((Self-Attention and Gaussian Attention) Graph Network (2S-AMAGN) based approach for ME classification in videos. The Self-Attention mechanism captures the global and local dependencies between nodes in a graph. The Gaussian attention mechanism computes weights based on the Gaussian distribution considering the mean and variance of features across each edge offering a nuanced understanding of spatial and temporal relationships within MEs. It meticulously analyzes node pair features and edge features capturing the significance of facial regions. An adaptive learnable weight is introduced to learn the contributions of each attention mechanism facilitating adaptive attention fusion. The network utilizes a three-frame graph structure to extract spatio-temporal information. The approach incorporates a dynamic frame selection mechanism which utilizes a sliding window optical flow method to filter out low-intensity emotion frames thereby refining the extraction of spatio-temporal information from the video data. The results are presented and compared with state-of-the-art methods for SMIC and SAMM databases. Additionally cross-dataset experiments are conducted and the results are reported.