Modality-specific Benchmarks and Radar Range-Doppler Envelope Classification for Multimodal Isolated Sign Language Recognition

Dmitriy Sazonov, Kamrul Islam, Evie Malaia, Sevgi Gurbuz; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 5046-5053

Recognition of isolated sign language (ISL) remains a challenged as many existing methods do not adequately account for the linguistic complexity in recorded data. From this perspective, radar is a sensing modality that has gained increased interest for ISL due to its unique ability to capture the dynamic, kinematic characteristics of sign language via measurement of Doppler shift. This paper proposes a novel method, RDM-Env, for classifying radar ISL data through computation of an efficient data representation derived from the upper and lower envelopes of range-Doppler map (RDM) videos. We benchmark radar-based and video-based models, as well as decision-level fusion paradigms against the MultiMediaLIS dataset provided for as part of the 1st ICCV Multimodal Isolated Italian Sign Language Recognition Challenge, which provides RGB video and radar RDM data. Our results show that both Uni-Sign (for video-based ISL) and RDM-Env (for radar-based ISL) surpass previous state-of-the-art models in their respective modalities for this dataset. Next, we evaluate several decision-level fusion paradigms, but observe that they degrade performance. Finally, we analyze modality-specific challenges with RDM-Env, finding that the majority of misclassifications were between fingerspelled signs.