Reducing Noise Pixels and Metric Bias in Semantic Inpainting on Segmentation Map

Jianfeng He, Bei Xiao, Xuchao Zhang, Shuo Lei, Shuhui Wang, Chang-Tien Lu; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 1876-1885

Semantic Inpainting on Segmentation Map (SISM) aims to manipulate segmentation maps by semantics. Providing structural assistance, segmentation maps have been broadly used as intermediate interfaces to achieve better image manipulation. We improve the SISM by considering the unique characteristics of segmentation maps in the both training and testing processes.First, to improve SISM training process, we reduce the noise pixels, which are pixel artifacts from the generation. Because each pixel in the segmentation maps has a much smaller value range in comparison to pixels in natural images, we propose a novel denoise activation (DA) by estimating the possible pixel values for an inpainted area in advance. Second, we improve SISM testing process by reducing the metric bias. The bias is caused by the ignore of latent ground truths in the current metrics in SISM. Based on the analysis of possible latent ground truths, we then propose a novel metric, Semantic Similarity (Sem), to quantify the semantic divergence between the generated and ground-truth target objects. Sem is calculated by a pre-trained semantic classifier using object shapes as training data. Since the classifier is pre-trained on PS-COCO dataset, with a large number of training samples and relatively general classes, Sem is also applicable to other datasets. Our experiments show impressive results of DA and Sem on three datasets.