Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation

Kurt H.W. Stolle; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 3301-3309

In this work we present Multiformer a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation monocular depth estimation and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation we propose a hybrid method using task-specific branches within each decoder block ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics outperforming previous methods by substantial margins. With a ResNet-50 backbone Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.