MambaPupil: Bidirectional Selective Recurrent Model for Event-based Eye Tracking

Zhong Wang, Zengyu Wan, Han Han, Bohao Liao, Yuliang Wu, Wei Zhai, Yang Cao, Zheng-Jun Zha; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5762-5770

Event-based eye tracking has shown great promise with the high temporal resolution and low redundancy provided by the event camera. However the diversity and abruptness of eye movement patterns including blinking fixating saccades and smooth pursuit pose significant challenges for eye localization. To achieve a stable event-based eye-tracking system this paper proposes a bidirectional long-term sequence modeling and time-varying state selection mechanism to fully utilize contextual temporal information in response to the variability of eye movements. Specifically the MambaPupil network is proposed which consists of the multi-layer convolutional encoder to extract features from the event representations a bidirectional Gated Recurrent Unit (GRU) and a Linear Time-Varying State Space Module (LTV-SSM) to selectively capture contextual correlation from the forward and backward temporal relationship. Furthermore the Bina-rep is utilized as a compact event representation and the tailor-made data augmentation called as Event-Cutout is proposed to enhance the model's robustness by applying spatial random masking to the event image. The evaluation of the ThreeET-plus benchmark shows that the MambaPupil realizes stable and accurate eye tracking under various complex conditions and achieves state-of-the-art performance.