Towards On-Device Learning on the Edge: Ways To Select Neurons To Update Under a Budget Constraint

Aël Quélennec, Enzo Tartaglione, Pavlo Mozharovskyi, Van-Tam Nguyen; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2024, pp. 685-694

In the realm of efficient on-device learning under extreme memory and computation constraints, a significant gap in successful approaches persists. Although a considerable effort has been devoted to making inference efficient, the primary hurdle for making training efficient centers around the prohibitive cost of backpropagation. The resource demands of computing gradients and updating network parameters often surpass the confines of tightly constrained memory budgets. This paper challenges the conventional wisdom and embarks on a series of experiments that reveal the existence of superior sub-networks. Additionally, we hint at the potential for substantial gains through a dynamic neuron selection strategy when fine-tuning a target task. Our efforts extend towards adapting a recent dynamic neuron selection strategy pioneered by Bragagnolo et al. (NEq), unveiling its effectiveness in the most stringent scenarios. Intriguingly, our experiments also demonstrate that a random selection approach outperforms dynamic neuron selection in less restrictive cases. This observation prompts a compelling avenue for further exploration, hinting at the need to develop a new class of algorithms designed to facilitate parameter update selection. Our findings usher in a new era of possibilities in the field of on-device learning under extreme constraints and encourage the pursuit of innovative strategies for efficient, resource-conscious model fine-tuning.