A Simple Finetuning Strategy Based on Bias-Variance Ratios of Layer-Wise Gradients

Mao Tomita, Ikuro Sato, Rei Kawakami, Nakamasa Inoue, Satoshi Ikehata, Masayuki Tanaka; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 471-487

Finetuning is an effective method for adapting pretrained networks to downstream tasks. However, the success of finetuning depends heavily on the selection of layers to be tuned, as full finetuning can lead to overfitting, while tuning only the last layer may not capture the necessary task-specific features. This requires a balanced approach of automatic layer selection to achieve higher performance. In this context, we propose the Bias-Variance Guided Layer Selection (BVG-LS), a simple yet effective strategy that adaptively selects a layer to be tuned at each training iteration. More specifically, BVG-LS computes the bias-variance ratios of mini-batch gradients for each layer and updates the parameters of the layer with the largest ratio. This strategy reduces the risk of overfitting while maintaining the model's capacity to learn task-specific features. In our experiments, we demonstrate the effectiveness of the BVG-LS strategy on seven image classification tasks. We show that BVG-LS outperforms full finetuning on all tasks with the WideResNet-50-2 model and on six out of seven tasks with the ViT-S model.