Low-Rank Adaptation vs. Fine-Tuning for Handwritten Text Recognition

Lukas Hüttner, Martin Mayr, Thomas Gorges, Fei Wu, Mathias Seuret, Andreas Maier, Vincent Christlein; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 1323-1332

The continuous expansion of neural network sizes is a notable trend in machine learning with transformer models exceeding 20 billion parameters in computer vision. This growth comes with rising demands for computational resources and large-scale datasets. Efficient techniques for transfer learning thus become an attractive option in setups with limited data as in handwriting recognition. Recently parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) and weight-decomposed low-rank adaptation (DoRA) have gained wide-spread interest. In this paper we explore trade-offs in parameter-efficient transfer learning using the synthetically pre-trained Transformer-Based Optical Character Recognition (TrOCR) model for handwritten text recognition with LoRA and DoRA. Additionally we analyze the performance of full fine-tuning with a limited number of samples scaling from a few-shot learning scenario up to using the whole dataset. We conduct experiments on the popular IAM Handwriting database as well as the historical READ 2016 dataset. We find that (a) LoRA/DoRA does not outperform full fine-tuning as opposed to a recent paper and (b) LoRA/DoRA is not substantially faster than full fine-tuning of TrOCR.