A Comprehensive Study of Transfer Learning Under Constraints

Tom Pégeot, Inna Kucher, Adrian Popescu, Bertrand Delezoide; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 1148-1157

Pre-training on an upstream task is widely used in deep learning to boost performance of downstream tasks. Recent studies analyzed pre-training with large datasets and large deep neural network architectures. However, pre-training is very useful in practice when downstream tasks have scarce data and are trained under computational constraints. To assess pre-training performance in this setting, we train different deep architectures with 1M parameters. We create different subsets of ImageNet to study the influence of upstream dataset in detail by varying the total size, but also the ratio between number of classes and samples per class for a constant total size. Then, we use the resulting models in transfer toward six diversified downstream tasks using linear probing and full fine tuning for downstream training. Experimental results confirm previous ones regarding performance saturation in downstream tasks, but we find that saturation occurs faster for compact deep architectures. The use of different ImageNet subsets leads to globally similar performance when enough data is included, regardless of the dataset structure. The comparison of downstream training strategies shows that linear probing can be competitive, particularly for few-shot settings. This is at odds with previous reports, which assert the superiority of full fine tuning. Finally, we observe that the type of deep architecture has a significant effect of results, but that their relative performance varies depending on the downstream training strategy.