SuperLoRA: Parameter-Efficient Unified Adaptation for Large Vision Models

Xiangyu Chen,Jing Liu,Ye Wang,Pu Wang,Matthew Brand,Guanghui Wang,Toshiaki Koike-Akino; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8050-8055

Abstract

Low-rank adaptation (LoRA) and its variants are widely employed in fine-tuning large models including large language models for natural language processing and diffusion models for computer vision. This paper proposes a generalized framework called SuperLoRA that unifies and extends different LoRA variants which can be realized under different hyper-parameter settings. Introducing new options with grouping folding shuffling projection and tensor decomposition SuperLoRA offers high flexibility and demonstrates superior performance with up to 10-fold gain in parameter efficiency for transfer learning tasks.

Related Material

[pdf] [supp]
[bibtex]
@InProceedings{Chen_2024_CVPR, author = {Chen, Xiangyu and Liu, Jing and Wang, Ye and Wang, Pu and Brand, Matthew and Wang, Guanghui and Koike-Akino, Toshiaki}, title = {SuperLoRA: Parameter-Efficient Unified Adaptation for Large Vision Models}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, month = {June}, year = {2024}, pages = {8050-8055} }