PD-GAN: Probabilistic Diverse GAN for Image Inpainting

Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 9371-9381

Abstract


We propose PD-GAN, a probabilistic diverse GAN forimage inpainting. Given an input image with arbitrary holeregions, PD-GAN produces multiple inpainting results withdiverse and visually realistic content. Our PD-GAN is builtupon a vanilla GAN which generates images based on random noise. During image generation, we modulate deepfeatures of input random noise from coarse-to-fine by injecting an initially restored image and the hole regions inmultiple scales. We argue that during hole filling, the pixels near the hole boundary should be more deterministic(i.e., with higher probability trusting the context and initially restored image to create natural inpainting boundary), while those pixels lie in the center of the hole shouldenjoy more degrees of freedom (i.e., more likely to dependon the random noise for enhancing diversity). To this end, we propose spatially probabilistic diversity normalization(SPDNorm) inside the modulation to model the probabilityof generating a pixel conditioned on the context information. SPDNorm dynamically balances the realism and diversity inside the hole region, making the generated content more diverse towards the hole center and resembleneighboring image content more towards the hole boundary. Meanwhile, we propose a perceptual diversity loss tofurther empower PD-GAN for diverse content generation. Experiments on benchmark datasets including CelebA-HQ, Places2 and Paris Street View indicate that PD-GAN is ef-fective for diverse and visually realistic image restoration.

Related Material


[pdf] [supp]
[bibtex]
@InProceedings{Liu_2021_CVPR, author = {Liu, Hongyu and Wan, Ziyu and Huang, Wei and Song, Yibing and Han, Xintong and Liao, Jing}, title = {PD-GAN: Probabilistic Diverse GAN for Image Inpainting}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2021}, pages = {9371-9381} }