DeepMIM: Deep Supervision for Masked Image Modeling

Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, Han Hu; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 879-888

Deep supervision which involves extra supervisions to the intermediate features of a neural network was widely used in image classification in the early deep learning era since it significantly reduces the training difficulty and eases the optimization like avoiding gradient vanish over the vanilla training. Nevertheless with the emergence of normalization techniques and residual connection deep supervision in image classification was gradually phased out. In this paper we revisit deep supervision for masked image modeling (MIM) that pre-trains a Vision Transformer (ViT) via a mask-and-predict scheme. Experimentally we find that deep supervision drives the shallower layers to learn more meaningful representations accelerates model convergence and expands attention diversities. Our approach called DeepMIM significantly boosts the representation capability of each layer. In addition DeepMIM is compatible with many MIM models across a range of reconstruction targets. For instance using ViT-B DeepMIM on MAE achieves 84.2 top-1 accuracy on ImageNet outperforming MAE by +0.6. By combining DeepMIM with a stronger tokenizer CLIP our model achieves state-of-the-art performance on various downstream tasks including image classification (85.6 top-1 accuracy on ImageNet-1K outperforming MAE-CLIP by +0.8) object detection (52.8 AP^box on COCO) and semantic segmentation (53.1 mIoU on ADE20K). Code and models are available at https://github.com/OliverRensu/DeepMIM.