Class-Aware Contrastive Learning for Fine-Grained Skeleton-Based Action Recognition

Xinyu Bian, Dongliang Chang, Yuqi Yang, Zhongjiang He, Kongming Liang, Zhanyu Ma; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3638-3654

Graph convolutional networks have significantly advanced skeleton-based action recognition by efficiently processing non-mesh skeleton sequences. However, existing methods struggle with fine-grained action recognition due to the high similarity of samples across categories. In this paper, we propose a class-aware contrastive learning framework designed to emphasize subtle motion feature differences. Our approach enhances discriminative capability for fine-grained action recognition by refining negative sample selection in contrastive learning to prioritize samples from similar categories. Furthermore, our framework incorporates global context from multiple sequences during the graph learning process and utilizes memory banks to store rich instance information, enriching cross-sequence context understanding. Our method achieves remarkable performance compared to state-of-the-art methods on the NTU RGB+D, NW-UCLA, and FineGym datasets. Codes are available at: https://github.com/PRIS-CV/Class-Aware-Contrastive-Learning-for-Action-Recognition.