Exemplar-Based Open-Set Panoptic Segmentation Network

Jaedong Hwang, Seoung Wug Oh, Joon-Young Lee, Bohyung Han; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 1175-1184

We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. The task requires to perform panoptic segmentation for not only known classes but also unknown ones that are not acknowledged during training. We investigate challenges of the task and present a benchmark dataset on top of an existing dataset, COCO. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies a new class with exemplars, which constructs pseudo-ground-truths, based on clustering and augments the size of each class by adding new exemplars based on their similarity during training. We evaluate the proposed method on our benchmark and demonstrate the effectiveness of our proposals. The goal of our work is to draw the attention of the community to the recognition in open-world scenarios.