Memory-Augmented Variational Adaptation for Online Few-Shot Segmentation

Jie Liu, Yingjun Du, Zehao Xiao, Cees G.M Snoek, Jan-Jakob Sonke, Efstratios Gavves; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 3324-3333

n this paper, we investigate online few-shot segmentation, which learns to make mask prediction for novel classes while observing samples sequentially. The main challenge in such an online scenario is the sample diversity in the sequence, resulting in models learned from previous samples that do not generalize well to future samples. To this end, we propose a memory-augmented variational adaptation network, which learns to adapt the model to each new sample that arrives sequentially. Specifically, we first introduce a contextual prototypical memory, which retains category knowledge from previous contextual information to facilitate the model adaptation to future samples. The adaptation to each new sample is then formulated as a variational Bayesian inference problem, which strives to generate sample-specific model parameters by conditioning the sample and the prototypical memory. Furthermore, we propose a feature customization module to learn sample-specific feature representation for better model adaptation to each sample in the sequence. With extensive experiments, we show that the proposed method effectively adapts to each sample from the online sample sequence, thus achieving state-of-the-art performance on both natural image and medical image datasets.