Forgetting-Free Incremental Panoptic Lifting by Maximum-Visibility Viewpoint Selection

Akira Kohjin, Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 7095-7104

We propose an incremental learning method for Panoptic Lifting, a novel view synthesis technique that leverages both RGB images and panoptic segmentation masks to represent a 3D scene. While Panoptic Lifting is valuable in applications such as VR, autonomous vehicles, and robotics, it faces a challenge when observations are limited (e.g., due to occlusions), as retraining from scratch becomes computationally expensive. Our approach incrementally updates the model to achieve high accuracy with reduced computational cost. To prevent catastrophic forgetting, a phenomenon where previous learned knowledge is lost during model updates, we introduce a viewpoint selection algorithm that solves a maximum coverage problem to identify a subset of viewpoints that maximizes the visibility of the scene. Experimental results demonstrate a 12% reduction in computation time compared to the naive approach, while effectively preventing catastrophic forgetting.