Decision-Only Adversarial Editing with Diffusion Models

Yaowen Wang, Daniel Cullina; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2026, pp. 1451-1460

Subtle visual cues, though often difficult for humans to localize, can critically affect the decisions of modern vision systems. Recent diffusion-based unrestricted adversarial examples have shown strong stealthiness and transferability, but most existing approaches assume white-box access or at least score-based feedback. In many real-world scenarios, however, only a hard-label decision (accept/reject) is available. We address this decision-only setting and propose AdvDiffEdit, a diffusion-guided method for adversarial generation and editing from a single binary oracle. AdvDiffEdit performs repeated SDEdit-style partial noising and denoising passes. In each pass, we (i) seed a worst-case forward noise at a chosen diffusion strength using a decision-only search, and (ii) progressively weaken guidance and strength across passes to "walk the data manifold" while preserving visual realism. Crucially, our method requires no gradients, logits, or scores---only one-bit feedback---yet reliably produces natural-looking decision-flipping images. Empirically, AdvDiffEdit achieves high attack success with strong visual fidelity and competitive cross-model transfer on ImageNet-class classifiers. Our results position diffusion-guided editing as a practical tool for auditing the robustness of decision-only systems to subtle, naturalistic visual perturbations under unrestricted threat models.