Information Theoretic Pruning of Coupled Channels in Deep Neural Networks

Peyman Rostami, Nilotpal Sinha, Nidhaleddine Chenni, Anis Kacem, Abd El Rahman Shabayek, Carl Shneider, Djamila Aouada; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 7765-7775

Variational channel pruning approaches have obtained impressive results thanks to their stochastic nature well established foundation in information theory and the practically appealing structured sparsity pattern they offer. Despite their success in pruning Plain Networks (PlainNets) their application has faced certain limitations in networks with structurally coupled channels such as ResNets. In such scenarios not only is it required to prune structurally coupled channels together but it is also necessary to ensure that the whole coupled group is irrelevant before pruning is applied. This is an under-investigated problem as most existing methods are designed without taking these couplings into account. In this paper we propose a novel approach based on Information Theoretic Pruning of structurally Coupled Channels (ITPCC) in neural networks. ITPCC allows for learning the probabilistic distribution of coupled channel set importance and prunes the ones withthe least relevant information to the task at hand. Experimental results for image classification on CIFAR10 CIFAR100 and ImageNet datasets show that the proposed method outperforms the state-of-the-art more significantly at high compression rates.