Towards Calibrated Gradient-based Multi-Task Learning

Linxiao Cao, Mianzimei Yang, Zhipeng Zhou, Hong Xie, Defu Lian, Menglin Yang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, 2026, pp. 5127-5136

Multi-task learning (MTL) enhances generalization and efficiency by jointly training multiple related tasks within a shared model. In recent years, numerous MTL paradigms have been proposed, among which gradient-based approaches have gained increasing popularity due to their strong performance and direct access to the optimization process. However, a key challenge in gradient-based MTL, i.e., gradient conflict, often hinders balanced learning. Importantly, we observe that this issue can be indeterministic due to gradient variance, which arises from factors such as small batch sizes, etc. In this paper, we provide the first study on how gradient variance influences the performance of gradient-based MTL methods. Our empirical analysis reveals that elevated gradient variance results in unstable updates and significantly impairs MTL performance. To mitigate this, we propose VarGrad, a lightweight and general framework that reduces gradient instability through iterative gradient correction and selectively schedules joint updates based on task-specific loss dynamics. Extensive experiments across multiple mainstream MTL benchmarks demonstrate that VarGrad achieves an average performance gain of 24.3%, while remaining broadly compatible with existing MTL methods.