ScatSimCLR: Self-Supervised Contrastive Learning With Pretext Task Regularization for Small-Scale Datasets

Vitaliy Kinakh, Olga Taran, Svyatoslav Voloshynovskiy; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 1098-1106

Abstract


In this paper, we consider a problem of self-supervised learning for small-scale datasets based on contrastive loss between multiple views of the data, which demonstrates the state-of-the-art performance in classification task. Despite the reported results, such factors as the complexity of training requiring complex architectures, the needed number of views produced by data augmentation, and their impact on the classification accuracy are understudied problems. To establish the role of these factors, we consider an architecture of contrastive loss system such as SimCLR, where baseline model is replaced by geometrically invariant "hard-crafted" network ScatNet with small trainable adapter network and argue that the number of parameters of the whole system and the number of views can be considerably reduced while practically preserving the same classification accuracy. In addition, we investigate the impact of regularization strategies using pretext task learning based on an estimation of parameters of augmentation transform such as rotation and jigsaw permutation for both traditional baseline models and ScatNet based models. Finally, we demonstrate that the proposed architecture with pretext task learning regularization achieves the state-of-the-art classification performance with a smaller number of trainable parameters and with reduced number of views.

Related Material


[pdf] [arXiv]
[bibtex]
@InProceedings{Kinakh_2021_ICCV, author = {Kinakh, Vitaliy and Taran, Olga and Voloshynovskiy, Svyatoslav}, title = {ScatSimCLR: Self-Supervised Contrastive Learning With Pretext Task Regularization for Small-Scale Datasets}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops}, month = {October}, year = {2021}, pages = {1098-1106} }