Channel Propagation Networks for Refreshable Vision Transformer

Junhyeong Go, Jongbin Ryu; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 1353-1362

In this paper we introduce the Channel Propagation method which aims to increase the channels of the Vision Transformer systematically. Skip connections are commonly acknowledged as a propagation approach that improves the stability of the performance in Vision Transformers. Nevertheless it is important to note that these skip connections may give rise to the problem of over-smoothing wherein similar features are represented in multiple layers. To tackle this matter our proposed approach for Channel Propagation in Vision Transformers retains the present signal information while concurrently propagating location-specific signals in a newly introduced channel dimension. On the other hand the proposed Channel Propagation method effectively maintains the integrity of identity representation while simultaneously including patch-wise location-specific supervision by introducing a new channel dimension. The inclusion of this approach in Vision Transformers mitigates the issue of over-smoothing while also improving the performance of visual recognition tasks. In our experiments we confirm that the proposed method is effective for various visual recognition tasks. Specifically our method demonstrates enhanced performance when implemented on Vision Transformer models; the classification accuracy is increased considerably for plain and hierarchical architectures on the ImageNet dataset.