Performance Evaluation of Segment Anything Model with Variational Prompting for Application to Non-Visible Spectrum Imagery

Yona Falinie A. Gaus, Neelanjan Bhowmik, Brian K. S. Isaac-Medina, Toby P. Breckon; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 3142-3152

The Segment Anything Model (SAM) is a deep neural network foundational model designed to perform instance segmentation which has gained significant popularity given its zero-shot segmentation ability.SAM operates by generating masks based on various input prompts such as text bounding boxes points or masks introducing a novel methodology to overcome the constraints posed by dataset-specific scarcity. While SAM is trained on an extensive dataset comprising more than 11M images it mostly consists of natural photographic (visible band) images with only very limited images from other modalities. Whilst the rapid progress in visual infrared surveillance and X-ray security screening imaging technologies driven forward by advances in deep learning has significantly enhanced the ability to detect classify and segment objects with high accuracy it is not evident if the SAM zero-shot capabilities can be transferred to such modalities beyond the visible spectrum. For this reason this work comprehensively assesses SAM capabilities in segmenting objects of interest in the X-ray and infrared imaging modalities. Our approach reuses and preserves the pre-trained SAM with three different prompts namely bounding box centroid and random points. We present several quantitative and qualitative results to showcase the performance of SAM on selected datasets. Our results show that SAM can segment objects in the X-ray modality when given a box prompt but its performance varies for point prompts. Specifically SAM performs poorly in segmenting slender objects and organic materials such as plastic bottles. Additionally we find that infrared objects are also challenging to segment with point prompts given the low-contrast nature of this modality. Overall this study shows that while SAM demonstrates outstanding zero-shot capabilities with box prompts its performance ranges from moderate to poor for point prompts indicating that special consideration on the cross-modal generalisation of SAM is needed when considering use on X-ray and infrared imagery.