Evaluation of Image Inpainting for Classification and Retrieval

Samuel Black, Somayeh Keshavarz, Richard Souvenir; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1060-1069

A common approach to censoring digital image content is masking the region(s) of interest with a solid color or pattern. In the case where the masked image will be used as input for classification or matching, the mask itself may impact the results. Recent work in image inpainting provides an alternative to masking by replacing the foreground with predicted background. In this paper, we perform an extensive evaluation of inpainting approaches to understand how well inpainted images can serve as proxies for the original in classification and retrieval. Results indicate that the metrics typically used to evaluate inpainting performance (e.g., reconstruction accuracy) do not necessarily correspond to improved classification or retrieval, especially in the case of person-shaped masked regions.