MultiPanoWise: Holistic Deep Architecture for Multi-task Dense Prediction from a Single Panoramic Image

Uzair Shah, Muhammad Tukur, Mahmood Alzubaidi, Giovanni Pintore, Enrico Gobbetti, Mowafa Househ, Jens Schneider, Marco Agus; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 1311-1321

We present a novel holistic deep-learning approach for multi-task learning from a single indoor panoramic image. Our framework named MultiPanoWise extends vision transformers to jointly infer multiple pixel-wise signals such as depth normals and semantic segmentation as well as signals from intrinsic decomposition such as reflectance and shading. Our solution leverages a specific architecture combining a transformer-based encoder-decoder with multiple heads by introducing in particular a novel context adjustment approach to enforce knowledge distillation between the various signals. Moreover at training time we introduce a hybrid loss scalarization method based on an augmented Chebychev/hypervolume scheme. We illustrate the capabilities of the proposed architecture on public-domain synthetic and real-world datasets. We demonstrate performance improvements with respect to the most recent methods specifically designed for single tasks like for example individual depth estimation or semantic segmentation. To our knowledge this is the first architecture capable of achieving state-of-the-art performance on the joint extraction of heterogeneous signals from single indoor omnidirectional images.