Cross-Task Attention Mechanism for Dense Multi-Task Learning

Ivan Lopes, Tuan-Hung Vu, Raoul de Charette; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 2329-2338

Abstract


Multi-task learning has recently become a promising solution for a comprehensive understanding of complex scenes. With an appropriate design multi-task models can not only be memory-efficient but also favour the exchange of complementary signals across tasks. In this work, we jointly address 2D semantic segmentation, and two geometry-related tasks, namely dense depth, surface normal estimation as well as edge estimation showing their benefit on indoor and outdoor datasets. We propose a novel multi-task learning architecture that exploits pair-wise cross-task exchange through correlation-guided attention and self-attention to enhance the average representation learning for all tasks. We conduct extensive experiments considering three multi-task setups, showing the benefit of our proposal in comparison to competitive baselines in both synthetic and real benchmarks. We also extend our method to the novel multi-task unsupervised domain adaptation setting. Our code is open-source.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Lopes_2023_WACV, author = {Lopes, Ivan and Vu, Tuan-Hung and de Charette, Raoul}, title = {Cross-Task Attention Mechanism for Dense Multi-Task Learning}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2023}, pages = {2329-2338} }