Explain with Confidence: Fusing Saliency Maps for Faithful and Interpretable Weakly-Supervised Model

Ayush Somani, Arif Ahmed Sekh, Dilip K. Prasad; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 290-299

Weakly supervised semantic segmentation (WSSS) faces the challenge of deriving accurate pixel-level masks from coarse image-level labels. While Explainable AI (XAI) methods can highlight class-discriminative regions, individual saliency maps often lack spatial completeness or consistency, limiting their utility for downstream tasks. We propose EWC (Explain with Confidence), a novel framework that fuses multiple explanation maps into a unified, interpretable signal for precise segmentation. At the core of EWC is a confidence-weighted fusion strategy that computes per-image explainer scores based on saliency concentration, inter-explainer agreement, and segmentation quality from a foundation model (SAM2). These scores are used to adaptively weight and combine saliency maps into reliable pseudo-labels. The fused maps are converted into reliable cues using a dual-thresholding strategy and passed as prompts to a frozen SAM2 model, producing final segmentation masks without any additional supervision or fine-tuning. Extensive experiments across five diverse benchmarks, including PASCAL VOC 2012 and CUB-200, demonstrate that our approach achieves competitive mIoU and provides faithful per-instance explanations over individual explainers. Our framework not only improves segmentation quality but also repositions explanation as a reliable training signal to foundation models for transparent weak supervision.