A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception

Asude Aydin, Mathias Gehrig, Daniel Gehrig, Davide Scaramuzza; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5701-5711

Spiking Neural Networks (SNNs) are a class of bio-inspired neural networks that promise to bring low-power and low-latency inference to edge-devices through the use of asynchronous and sparse processing. However being temporal models SNNs depend heavily on expressive states to generate predictions on par with classical artificial neural networks (ANNs). These states converge only after long transient time periods and quickly decay in the absence of input data leading to higher latency power consumption and lower accuracy. In this work we address this issue by initializing the state with an auxiliary ANN running at a low rate. The SNN then uses the state to generate predictions with high temporal resolution until the next initialization phase. Our hybrid ANN-SNN model thus combines the best of both worlds: It does not suffer from long state transients and state decay thanks to the ANN and can generate predictions with high temporal resolution low latency and low power thanks to the SNN. We show for the task of event- based 2D and 3D human pose estimation that our method consumes 88% less power with only a 4% decrease in performance compared to its fully ANN counterparts when run at the same inference rate. Moreover when compared to SNNs our method achieves a 74% lower error. This research thus provides a new understanding of how ANNs and SNNs can be used to maximize their respective benefits.