Opportunities for Post-Training Dynamic Layer Sparsity in Large Vision and Language Models

Jordan Dotzel, Carly Jiang, Mohamed Abdelfattah, Zhiru Zhang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8280-8284

Large language and vision models have recently achieved state-of-the-art performance across various tasks yet due to their large computational requirements they struggle with strict memory latency and power demands. To meet these demands various forms of dynamic sparsity have been proposed that reduce compute on an input-by-input basis. These methods improve over static methods by exploiting the variance across individual inputs which has steadily grown with the exponential increase in training data. This dynamic sparsity has been explored within the hidden dimension and attention heads. Yet the increasing depth within modern models currently with hundreds of layers has opened opportunities for dynamic layer sparsity which skips the computation for entire layers. In this work we explore the practicality of layer sparsity within pre-trained models by profiling residual connections and establish the relationship between model depth and layer sparsity. For example the residual blocks in the OPT-66B model have a median contribution of 5% to its output and ViT-Huge has approximately a 7% contribution. We also find these contributions decrease linearly with model size implying that state-of-the-art models have near a 1% median contribution on each layer which creates significant opportunities for dynamic layer sparsity. We then insert oracles at each layer and threshold on these residual contributions to find that these models can support significant dynamic sparsity with median dynamic depth close to 75% of their original depth.