CATS: Combined Activation and Temporal Suppression for Efficient Network Inference

Zeqi Zhu, Arash Pourtaherian, Luc Waeijen, Ibrahim Batuhan Akkaya, Egor Bondarev, Orlando Moreira; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 8166-8175

Brain-inspired event-driven processors execute deep neural networks (DNNs) in an event sparsity-aware manner, leading to superior performance compared to conventional platforms. In the pursuit of higher event sparsity, prior studies suppress non-zero events by either eliminating the intra-frame activations (spatially) or leveraging the redundancy in the inter-frame differences for a video (temporally). However, we have empirically observed that simultaneously enhancing activation and temporal sparsity can lead to a synergistic suppression outcome. To this end, we propose an end-to-end event suppression training approach CATS -- Combined Activation and Temporal Suppression for efficient network inference. It utilizes a gradient-based method to search for the optimal temporal thresholds for the network while penalizing the presence of non-zero events in spatial and temporal domains simultaneously. We demonstrate that CATS achieves 2 6 times more event suppression compared to the inherent ReLU suppression, consistently outperforming the SOTA by a significant margin at various accuracy levels. Extensive experimental results show that CATS also generalizes to multiple tasks -- object detection, object tracking, pose estimation, and semantic segmentation. Furthermore, a case study for the commercial event-driven processor GrAI-VIP highlights that the induced event sparsity in SSD on EgoHands datasets efficiently translates into significant improvements of 2.5 x in FPS, 2.1 x in latency, and 3.8 x in energy consumption, while maintaining the model accuracy.