Gate-Shift-Pose: Enhancing Action Recognition in Sports with Skeleton Information

Edoardo Bianchi, Oswald Lanz; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 1257-1264

This paper introduces Gate-Shift-Pose an enhanced version of Gate-Shift-Fuse networks designed for athlete fall classification in figure skating by integrating skeleton pose data alongside RGB frames. We evaluate two fusion strategies: early-fusion which combines RGB frames with Gaussian heatmaps of pose keypoints at the input stage and late-fusion which employs a multi-stream architecture with attention mechanisms to combine RGB and pose features. Experiments on the FR-FS dataset demonstrate that Gate-Shift-Pose significantly outperforms the RGB-only baseline improving accuracy by up to 40% with ResNet18 and 20% with ResNet50. Early-fusion achieves the highest accuracy (98.08%) with ResNet50 leveraging the model's capacity for effective multimodal integration while late-fusion is better suited for lighter backbones like ResNet18. These results highlight the potential of multimodal architectures for sports action recognition and the critical role of skeleton pose information in capturing complex motion patterns.