Dynamic Knowledge Adapter with Probabilistic Calibration for Generalized Few-Shot Semantic Segmentation

Jintao Tong, Haichen Zhou, Yicong Liu, Yiman Hu, Yixiong Zou; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2781-2790

Generalized Few-shot Semantic Segmentation (GFSS) aims to use a few novel-class samples to enable the model trained on base classes to have the ability to segment for all classes (including base and novel classes). We analyze the three main reasons for the model's limited performance on GFSS: the lack of adaptability to learn novel classes the instability that causes the catastrophic forgetting of base classes and the biased prediction of imbalanced classes. To handle these issues we design an auxiliary network (Dynamic Knowledge Adapter DKA) for the GFSS task. Firstly DKA handles the adaptability problem by selecting only efficient parameters for finetuning. Secondly DKA addresses the stability problem by relabelling part of the training samples for iterative training which alleviates the conflict between base and novel classes. Thirdly it involves a probabilistic calibration module to help the model rectify the prediction bias caused by imbalanced data. Experimental results show that these designs can help the model to take into account the segmentation performance of base classes novel classes and the background class that is to perform well in all-class segmentation.