SAMIF: Adapting Segment Anything Model for Image Inpainting Forensics

Lan Zhang, Xinshan Zhu, Di He, Xin Liao, Biao Sun; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3605-3621

Image inpainting technologies pose increasing threats to the security of image data through malicious use. Therefore, image inpainting forensics is crucial. The Segment Anything Model (SAM) is a powerful universal image segmentation model for various downstream tasks. However, the performance of SAM in inpainting forensics is significantly degraded due to the substantial disparity between natural and inpainted image domains. In this paper, we propose SAMIF, a SAM-based model for image inpainting forensics. First, based on SAM, a parallel convolutional neural network (CNN) branch is introduced to assist the SAM in extracting local noise information. Second, the cross-domain alignment fusion module (CAFM) is designed to better fuse the features of the two branches. Third, the artifact features generator (AFG) is designed between the encoder and decoder to disentangle the features extracted by the encoder. The auxiliary loss is introduced in AFG, which shortens the backpropagation path and guides the SAM branch to learn artifact features, thus enhancing the adaptability of SAM for the inpainting forensics task. Extensive experiments demonstrate that the proposed model achieves state-of-the-art results on five inpainting forensics datasets and exhibits excellent robustness and generalization capabilities.