Enrich Distill and Fuse: Generalized Few-Shot Semantic Segmentation in Remote Sensing Leveraging Foundation Model's Assistance

Tianyi Gao, Wei Ao, Xing-Ao Wang, Yuanhao Zhao, Ping Ma, Mengjie Xie, Hang Fu, Jinchang Ren, Zhi Gao; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2771-2780

Generalized few-shot semantic segmentation (GFSS) unifies semantic segmentation with few-shot learning showing great potential for Earth observation tasks under data scarcity conditions such as disaster response urban planning and natural resource management. GFSS requires simultaneous prediction for both base and novel classes with the challenge lying in balancing the segmentation performance of both. Therefore this paper introduces a novel framework named FoMA Foundation Model Assisted GFSS framework for remote sensing images. We aim to leverage the generic semantic knowledge inherited in foundation models. Specifically we employ three strategies named Support Label Enrichment (SLE) Distillation of General Knowledge (DGK) and Voting Fusion of Experts (VFE). For the support images SLE explores credible unlabeled novel categories ensuring that each support label contains multiple novel classes. For the query images DGK technique allows an effective transfer of generalizable knowledge of foundation models on certain categories to the GFSS learner. Additionally VFE strategy integrates the zero-shot prediction of foundation models with the few-shot prediction of GFSS learners achieving improved segmentation performance. Extensive experiments and ablation studies conducted on the OpenEarthMap few-shot challenge dataset demonstrate that our proposed method achieves state-of-the-art performance.