Enhancing the Reliability of Auto-Prompting SAM for Medical Image Segmentation with Uncertainty Estimation and Rectification

Yichi Zhang, Shiyao Hu, Le Xue, Sijie Ren, Zixin Hu, Yuan Cheng, Yuan Qi; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 1282-1291

Automatic segmentation of medical images has widespread applications in modern clinical workflows. Recent advancements in prompt-driven foundation models have shown remarkable potential for universal medical image segmentation without the need for specific domain training. However, their reliance on prompts necessitates human-computer interaction during the inference process, which directly increases the burden for applications. Auto-prompting methods have been introduced to enable automatic segmentation but often lack reliability, as low-quality prompts may significantly compromise the accuracy of segmentation. To enhance the reliability of auto-prompting for medical image segmentation based on foundation models, we introduce a prompts-triggered uncertainty estimation strategy to evaluate voxel-level reliability of model-generated segmentation. Based on the estimated uncertainty, we design a simple yet efficient rectification strategy to automatically refine possible mistakes with high uncertainty and improve the segmentation accuracy. We conducted quantitative and qualitative assessments on the performance of our proposed method on two representative medical datasets covering the segmentation task of 22 head-and-neck organs and 13 abdominal organs. Experimental results demonstrate uncertainty rectification obtains significant improvements in dice similarity coefficient with up to 10.7 % and 13.8 %, surpassing other comparative methods. The uncertainty map not only improves the final segmentation result with the rectification strategy but also enables the identification of potential segmentation errors and supports further analysis, offering valuable guidance in areas where manual focus and refinement are required for clinicians.