Continuous Gesture Recognition With Hand-Oriented Spatiotemporal Feature

Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 3056-3064

In this paper, an efficient spotting-recognition framework is proposed to tackle the large scale continuous gesture recognition problem with the RGB-D data input. Concretely, continuous gestures are firstly segmented into isolated gestures based on the hand positions obtained by our proposed two streams Faster R-CNN. In the subsequent recognition stage, firstly, a specific hand-oriented spatiotemporal feature is extracted by 3D convolutional network. In this feature, only the hand regions and face location are considered, which can effectively block the negative influence of the distractors. Next, the extracted features from RGB and depth are fused to boost the representative power and the classification is achieved by using the linear SVM. Extensive experiments are conducted to validate the effectiveness of the proposed method. Our method achieves the mean Jaccard Index of 0.6103 and outperforms other results in the ChaLearn LAP Large-scale Continuous Gesture Recognition Challenge.