Exploring Human-Model Alignment in Visual Social Attention During Help-and-Hinder Social Interaction Classification

Lucia Schiatti, Guido Vallarino, Sabrina Megan Lopez, Yen-Ling Kuo, Matteo Moro, Mengmi Zhang, Monica Gori, Alessio Del Bue, Boris Katz, Andrei Barbu; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 4913-4922

Artificial Neural Networks (ANNs) are considered the best models of human vision in many visual tasks. However, there is still a significant gap between humans and ANNs in visual tasks involving social perception and cognition. In this paper, we investigate the alignment between humans and different ANNs architectures, i.e., convolutional neural networks (CNNs) and transformers, during the classification of social interactions from videos. Specifically, we provide a novel dataset of videos and human gaze data during the classification of help and hinder social interactions, and we evaluate the human-model alignment in terms of classification accuracy and visual attention. We show that our proposed dataset and experimental protocol can enable comparison between different models' alignment to humans in terms of both static and dynamic visual attention. Our results suggest that a higher classification accuracy may be correlated with a higher similarity of the model's attention to human gaze-based saliency maps. However, a stronger alignment of human-model attention is not always predictive of a better classification.