PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering

Yurui Ren, Ge Li, Yuanqi Chen, Thomas H. Li, Shan Liu; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 13759-13768

Abstract


Generating portrait images by controlling the motions of existing faces is an important task of great consequence to social media industries. For easy use and intuitive control, semantically meaningful and fully disentangled parameters should be used as modifications. However, many existing techniques do not provide such fine-grained controls or use indirect editing methods i.e. mimic motions of other individuals. In this paper, a Portrait Image Neural Renderer (PIRenderer) is proposed to control the face motions with the parameters of three-dimensional morphable face models (3DMMs). The proposed model can generate photo-realistic portrait images with accurate movements according to intuitive modifications. Experiments on both direct and indirect editing tasks demonstrate the superiority of this model. Meanwhile, we further extend this model to tackle the audio-driven facial reenactment task by extracting sequential motions from audio inputs. We show that our model can generate coherent videos with convincing movements from only a single reference image and a driving audio stream. Our source code is available at https://github.com/RenYurui/PIRender.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Ren_2021_ICCV, author = {Ren, Yurui and Li, Ge and Chen, Yuanqi and Li, Thomas H. and Liu, Shan}, title = {PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {13759-13768} }