From 2D Portraits to 3D Realities: Advancing GAN Inversion for Enhanced Image Synthesis

Wonseok Oh, Youngjoo Jo; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 737-746

Image synthesis using StyleGAN has shown remarkable results in 2D portrait image generation. The works of the GAN inversion to manipulate the real image using StyleGAN latent space also show remarkable achievements. 2D GAN inversion has successfully manipulated global attributes such as facial expressions and gender. However preserving the hairstyle and identity was difficult according to the pose change. We introduce the 3D GAN inversion encoder to make a high-resolution 3D image based on the Geometry Aware 3D Generative Adversarial Network known as EG3D which allows explicit control over the pose of the real image subject with multi-view consistency. Our network projects the single 2D portrait images to novel latent space for 3D GAN inversion for the tri-plane of EG3D. We also present multi-view cycle loss which aims to increase multi-view consistency. By leveraging the new latent space and loss for 3D GAN inversion our network can successfully convert 2D portrait images into 3D fast.