Repetition-Aware Image Sequence Sampling for Recognizing Repetitive Human Actions

Konstantinos Bacharidis, Antonis Argyros; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 1878-1887

In the field of video-based human action recognition (HAR), standard hand-crafted and deep learning-based approaches are constrained by the computational and memory requirements of their models and the length of the input sequence that can be processed during learning. Sampling techniques employing a windowed or a random clip cropping have been the simplest and most effective ways to cope with limitations on the maximum possible length of the input sequence. However, such designs do not guarantee that the correct ordering of the action steps is captured, or require several learning iterations. In this work we address this problem for the class of repetitive actions. Specifically, given a temporal segmentation of a repetitive action into its repetitive segments, we propose and develop novel approaches for ranking and selecting/sampling segments so as to improve learning in deep models for HAR. We show that by employing the proposed repetition-aware sampling schemes in state-of-the-art deep models for HAR, the action recognition accuracy is increased. The proposed approach is evaluated on existing datasets and on a new dataset that is tailored to the quantitative evaluation of the task at hand. The obtained results reveal how our approach performs in relation to various characteristics of the observed repetitive actions (repetition frequency, their effects on sscene objects, etc) and demonstrate the performance improvements.