Neuromorphic Lip-Reading with Signed Spiking Gated Recurrent Units

Manon Dampfhoffer, Thomas Mesquida; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2141-2151

Automatic Lip-Reading (ALR) requires the recognition of spoken words based on a visual recording of the speaker's lips without access to the sound. ALR with neuromorphic event-based vision sensors instead of traditional frame-based cameras is particularly promising for edge applications due to their high temporal resolution low power consumption and robustness. Neuromorphic models such as Spiking Neural Networks (SNNs) encode information using events and are naturally compatible with such data. The sparse and event-based nature of both the sensor data and SNN activations can be leveraged in an end-to-end neuromorphic hardware pipeline for low-power and low-latency edge applications. However the accuracy of SNNs is often largely degraded compared to state-of-the-art non-spiking Artificial Neural Networks (ANNs). In this work a new SNN model the Signed Spiking Gated Recurrent Unit (SpikGRU2+) is proposed and used as a task head for event-based ALR. The SNN architecture is as accurate as its ANN equivalent and outperforms the state-of-the-art on the DVS-Lip dataset. Notably the accuracy is improved by 25% (respectively 4%) compared to the previous state-of-the-art SNN (respectively ANN). In addition the SNN spike sparsity can be optimized to further reduce the number of operations up to 22x compared to the ANN while maintaining a high accuracy. This work opens up new perspectives for the use of SNNs for accurate and low-power end-to-end neuromorphic gesture recognition. Code is available.