Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision

Tarun Kalluri, Weiyao Wang, Heng Wang, Manmohan Chandraker, Lorenzo Torresani, Du Tran; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2693-2703

Top-down instance segmentation architectures excel with predefined closed-world taxonomies but exhibit biases and performance degradation in open-world scenarios. In this work we introduce bottom-Up and top-Down Open-world Segmentation (UDOS) a novel approach that combines classical bottom-up segmentation methods within a top-down learning framework. UDOS leverages a top-down network trained with weak supervision derived from class-agnostic bottom-up segmentation to predict object parts. These part-masks undergo affinity-based grouping and refinement to generate precise instance-level segmentations. UDOS balances the efficiency of top-down architectures with the capacity to handle unseen categories through bottom-up supervision. We validate UDOS on challenging datasets (MS-COCO LVIS ADE20k UVO and OpenImages) achieving superior performance over state-of-the-art methods in cross-category and cross-dataset transfer tasks. Our code and models will be publicly available.