Body Pose Sonification for a View-Independent Auditory Aid to Blind Rock Climbers

Joseph Ramsay, Hyung Jin Chang; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 3414-3421

Rock climbing is a sport in which blind people have traditionally found it extremely difficult to excel due to the high degree of visual problem solving required, and also the requirement to climb with a sighted assistant. We present a system which automates the role of the sighted assistant in order to provide blind people with the freedom to climb and train on their own. We address climbing-specific limitations of a state-of-the-art skeleton tracking system, and discuss the ways in which we mitigated these limitations using post-processing techniques tuned specially for a climbing scenario. We also describe the auditory feedback system used to instruct the blind climber, and demonstrate that a user can learn to follow it in a relatively short time by showing a significant improvement in performance over just a few trials with the system.