AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning

Guoxian Song, Hongyi Xu, Jing Liu, Tiancheng Zhi, Yichun Shi, Jianfeng Zhang, Zihang Jiang, Jiashi Feng, Shen Sang, Linjie Luo; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 765-774

While substantial progresses have been made in automated 2D portrait stylization admirable 3D portrait stylization from a single user photo remains to be an unresolved challenge. One primary obstacle here is the lack of high quality stylized 3D training data. In this paper we propose a novel framework AgileGAN3D that can produce 3D artistically appealing and personalized portraits with detailed geometry. New stylization can be obtained with just a few (around 20) unpaired 2D exemplars. We achieve this by first leveraging existing 2D stylization capabilities style prior creation to produce a large amount of augmented 2D style exemplars. These augmented exemplars are generated with accurate camera pose labels as well as paired real face images which prove to be critical for the downstream 3D stylization task. Capitalizing on the recent advancement of 3D-aware GAN models we perform guided transfer learning on a pretrained 3D GAN generator to produce multi-view-consistent stylized renderings. In order to achieve 3D GAN inversion that can preserve subject's identity well we incorporate multi-view consistency loss in the training of our encoder. Our pipeline demonstrates strong capability in turning user photos into a diverse range of 3D artistic portraits. Both qualitative results and quantitative evaluations have been conducted to show the superior performance of our method. Code and pretrained models will be released for reproduction purpose.