RF-ChessSIGN: Radar-enabled Human-Computer Interaction in a Real-Time Sign Language-Controlled Game

Kenneth Dehaan, Emre Kurtoglu, Sabyasachi Biswas, Caroline Kobek Pezzarossi, Darrin Griffin, Chris Crawford, Ali Gurbuz, Evie Malaia, Abraham Glasser, Raja Kushalnagar, Sevgi Zubeyde Gurbuz; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 5000-5010

Radio frequency (RF) sensors have gained an enormous popularity in the past decade with advances in machine learning and small-package, high-sensitivity sensor design. Cost efficiency and robustness against environmental factors such as lighting, skin color or room layout make them ideal sensors for indoor monitoring applications such as human activity, hand gesture and sign language recognition. Although there has been great interest in the research community in exploring the potential of radars for sign language recognition, most studies are often conducted in controlled laboratory environments with certain experimental restrictions and instructions limiting the user behavior, which consequently yield overoptimistic results. Moreover, these studies often consist of repetitive articulation of the signs of interest, which is overwhelming for the participants. In this work, we explore the real-world challenges of RF sensors by freeing the data collection process from experimental restrictions by gamifying it. Essentially, we develop and deploy a radar-based sign language-controlled Chess game (RF-ChessSIGN) where users move the Chess pieces on the board by articulating the signs registered to a particular position. The developed approach has enabled an entertaining and sustainable way of collecting data, and revealed the ineffectiveness of the datasets acquired under controlled experimental settings which cannot capture the nuances and intricacies of natural signing.