PP-SAM: Perturbed Prompts for Robust Adaption of Segment Anything Model for Polyp Segmentation

Md Mostafijur Rahman, Mustafa Munir, Debesh Jha, Ulas Bagci, Radu Marculescu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 4989-4995

The Segment Anything Model (SAM) originally designed for general-purpose segmentation tasks has been used recently for polyp segmentation. Nonetheless fine-tuning SAM with data from new imaging centers or clinics poses significant challenges. This is because this necessitates the creation of an expensive and time-intensive annotated dataset along with the potential for variability in user prompts during inference. To address these issues we propose a robust fine-tuning technique PP-SAM that allows SAM to adapt to the polyp segmentation task with limited images. To this end we utilize variable perturbed bounding box prompts (BBP) to enrich the learning context and enhance the model's robustness to BBP perturbations during inference. Rigorous experiments on polyp segmentation benchmarks reveal that our variable BBP perturbation significantly improves model resilience. Notably on Kvasir 1-shot fine-tuning boosts the DICE score by 20% and 37% with 50 and 100-pixel BBP perturbations during inference respectively. Moreover our experiments show that 1-shot 5-shot and 10-shot PP-SAM with 50-pixel perturbations during inference outperform a recent state-of-the-art (SOTA) polyp segmentation method by 26% 7% and 5% DICE scores respectively. Our results motivate the broader applicability of our PP-SAM for other medical imaging tasks with limited samples. Our implementation is available at https://github.com/SLDGroup/PP-SAM.