Towards Quantitative Evaluation Metrics for Image Editing Approaches

Dana Cohen Hochberg,Oron Anschel,Alon Shoshan,Igor Kviatkovsky,Manoj Aggarwal,Gerard Medioni; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 7892-7900

In the rapidly evolving field of Generative AI this work takes initial steps towards establishing a systematic approach for comparing image editing methods. Currently there is a lack of quantitative metrics for evaluating image editing tasks with new methods being evaluated mostly qualitatively. Our methodology involves three key components: 1) The creation of a large synthetic dataset using GAN-Control which enables the generation of ground-truth images for consistent edits across different facial identities; 2) A matching procedure that pairs the edited images with their corresponding ground-truth; and 3) Application of the Perceptual Distance metric to matched pairs. We assessed the effectiveness of our proposed framework through a user study and a set of simulation experiments. Our results indicate that our approach can rank image-editing methods in a way that aligns with human judgment. This research seeks to lay the foundation for a comprehensive evaluation framework for image editing techniques in subsequent studies initiating a dialogue on this topic.