STAR: Sparse Thresholded Activation Under Partial-Regularization for Activation Sparsity Exploration

Zeqi Zhu, Arash Pourtaherian, Luc Waeijen, Egor Bondarev, Orlando Moreira; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 4554-4563

Brain-inspired event-driven processors execute deep neural networks (DNNs) in a sparsity-aware manner. Specifically, if more zeros are induced in the activation maps, less computation will be performed in the succeeding convolution layer. However, inducing activation sparsity in DNNs remains a challenge. To address this, we propose a training approach STAR (Sparse Thresholded Activation under partial-Regularization), which combines activation regularization with thresholding, to overcome the barrier of a single threshold- or regularization-based method in sparsity improvement. More precisely, we employ the sparse penalty on the near-zero activations to fit the activation learning behavior in accuracy recovery, followed by thresholding to further suppress activations. Experimental results with SOTA networks (ResNet50/MobileNetV2, SSD, YOLOX and DeepLabV3+) on various datasets (ImageNet, KITTI, VOC2007 and CityScapes) show that STAR can reduce on average 54% more activations compared to ReLU suppression. It outperforms the state-of-the-art by a significant margin of 35% in activation suppression without compromising accuracy loss. Additionally, a case study for a commercially-available event-driven hardware architecture, Neuronflow, demonstrates that the boosted activation sparsity in ResNet50 can be efficiently translated into latency reduction by up to 2.78x, FPS improvement by up to 2.80x, and energy savings by up to 2.09x. STAR elevates event-driven processors as a superior alternative to GPUs for Edge computing.