HNSSL: Hard Negative-Based Self-Supervised Learning

Wentao Zhu, Jingya Liu, Yufang Huang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 4778-4787

Recently, learning from vast unlabeled data, especially self-supervised learning, has been emerging and attracting widespread attention. Self-supervised learning followed by supervised fine-tuning on a few labeled examples can significantly improve label efficiency and outperform standard supervised training using fully annotated data. In this work, we present a novel hard negative-based self-supervised deep learning paradigm, named HNSSL. Specifically, we design a student-teacher network to generate a multi-view of the data for self-supervised learning and integrate an online hard negative pair mining into the training. Then we derive a new triplet-type loss considering both positive sample pairs and online mined hard negative sample pairs. Extensive experiments demonstrate the effectiveness of the proposed method and its components on ILSVRC-2012 based on the same backbone network. Specifically, for the linear evaluation task, the proposed HNSSL with a ResNet-50 encoder achieves the top-1 accuracy of 77.1%, which outperforms its previous counterparts by 2.8%. For the semi-supervised learning task, HNSSL with a ResNet-50 encoder obtains the top-1 accuracy of 73.4%, which outperforms the previous ResNet-50 encoder-based semi-supervised learning results by 4.6% using only 10% labels. For the task of transfer learning with linear evaluation, HNSSL with a ResNet-50 encoder achieves the best accuracy on six of seven widely used transfer learning datasets, which averagely outperforms previous ResNet-50 encoder-based transfer learning results by 2.5%.