Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-The-Wild

Shuang Liu, Zhao Wang, Xiaosong Yang, Jianjun Zhang; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 777-785

With the increasing amount of videos recorded using 2D mobile cameras, the technique for recovering the 3D dynamic facial models from these monocular videos has become a necessity for many image and video editing applications. While methods based parametric 3D facial models can reconstruct the 3D shape in dynamic environment, large structural changes are ignored. Structure-from-motion methods can reconstruct these changes but assume the object to be static. To address this problem we present a novel method for realtime dynamic 3D facial tracking and reconstruction from videos captured in uncontrolled environments. Our method can track the deforming facial geometry and reconstruct external objects that protrude from the face such as glasses and hair. It also allows users to move around, perform facial expressions freely without degrading the reconstruction quality.