Towards Automated Understanding of Student-Tutor Interactions using Visual Deictic Gestures

Suchitra Sathayanarayana, Ravi Kumar Satzoda, Amber Carini, Monique Lee, Linda Salamanca, Judy Reilly, Deborah Forster, Marian Bartlett, Gwen Littlewort; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2014, pp. 474-481

Abstract


In this paper, we present techniques for automated understanding of tutor-student behavior through detecting visual deictic gestures, in the context of one-to-one mathematics tutoring. To the best knowledge of the authors, this is the first work in the area of intelligent tutoring systems, which focuses on spatial localization of deictic gestural activity, i.e. where the deictic gesture is pointing on the workspace. A new dataset called SDMATH is first introduced. The motivation for detecting deictic gestures and their spatial properties is established, followed by techniques for automatic localization of deictic gestures in a workspace. The techniques employ computer vision and machine learning steps such as GBVS saliency, binary morphology and HOG-SVM classification. It is shown that the method localizes the deictic tip with an accuracy of over 85% accuracy for a cut off distance of 12 pixels. Furthermore, a detailed discussion using examples from the proposed dataset is presented on high-level inferences about the student-tutor interactions that can be derived from the integration of spatial and temporal localization of the deictic gestural activity using the proposed techniques.

Related Material


[pdf]
[bibtex]
@InProceedings{Sathayanarayana_2014_CVPR_Workshops,
author = {Sathayanarayana, Suchitra and Kumar Satzoda, Ravi and Carini, Amber and Lee, Monique and Salamanca, Linda and Reilly, Judy and Forster, Deborah and Bartlett, Marian and Littlewort, Gwen},
title = {Towards Automated Understanding of Student-Tutor Interactions using Visual Deictic Gestures},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2014}
}