Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing

Ling Lo, Cheng Yu Yeo, Hong-Han Shuai, Wen-Huang Cheng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 24462-24471

Abstract


Recent text-to-image (T2I) diffusion models have revolutionized image editing by empowering users to control outcomes using natural language. However the ease of image manipulation has raised ethical concerns with the potential for malicious use in generating deceptive or harmful content. To address the concerns we propose an image immunization approach named semantic attack to protect our images from being manipulated by malicious agents using diffusion models. Our approach focuses on disrupting the semantic understanding of T2I diffusion models regarding specific content. By attacking the cross-attention mechanism that encodes image features with text messages during editing we distract the model's attention regarding the content of our concern. Our semantic attack renders the model uncertain about the areas to edit resulting in poorly edited images and contradicting the malicious editing attempts. In addition by shifting the attack target towards intermediate attention maps from the final generated image our approach substantially diminishes computational burden and alleviates GPU memory constraints in comparison to previous methods. Moreover we introduce timestep universal gradient updating to create timestep-agnostic perturbations effective across different input noise levels. By treating the full diffusion process as discrete denoising timesteps during the attack we achieve equivalent or even superior immunization efficacy with nearly half the memory consumption of the previous method. Our contributions include a practical and effective approach to safeguard images against malicious editing and the proposed method offers robust immunization against various image inpainting and editing approaches showcasing its potential for real-world applications.

Related Material


[pdf] [supp]
[bibtex]
@InProceedings{Lo_2024_CVPR, author = {Lo, Ling and Yeo, Cheng Yu and Shuai, Hong-Han and Cheng, Wen-Huang}, title = {Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {24462-24471} }