Low-Rank Few-Shot Adaptation of Vision-Language Models

Maxime Zanella, Ismail Ben Ayed; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 1593-1603

Recent progress in the few-shot adaptation of Vision-Language Models (VLMs) has further pushed their generalization capabilities at the expense of just a few labeled samples within the target downstream task. However this promising already quite abundant few-shot literature has focused principally on prompt learning and to a lesser extent on adapters overlooking the recent advances in Parameter-Efficient Fine-Tuning (PEFT). Furthermore existing few-shot learning methods for VLMs often rely on heavy training procedures and/or carefully chosen task-specific hyper-parameters which might impede their applicability. In response we introduce Low-Rank Adaptation (LoRA) in few-shot learning for VLMs and show its potential on 11 datasets in comparison to current state-of-the-art prompt- and adapter-based approaches. Surprisingly our simple CLIP-LoRA method exhibits substantial improvements while reducing the training times and keeping the same hyper-parameters in all the target tasks i.e. across all the datasets and numbers of shots. Certainly our surprising results do not dismiss the potential of prompt-learning and adapter-based research. However we believe that our strong baseline could be used to evaluate progress in these emergent subjects in few-shot VLMs.