Disjoint Pose and Shape for 3D Face Reconstruction

Raja Kumar, Jiahao Luo, Alex Pang, James Davis; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 3115-3125

Existing methods for 3D face reconstruction from a few casually captured images employ deep learning based models along with a 3D Morphable Model(3DMM) as face geometry prior. Structure From Motion(SFM), followed by Multi-View Stereo (MVS), on the other hand, uses dozens of high-resolution images to reconstruct accurate 3D faces. However, it produces a noisy result with only two views. In this paper, taking inspiration from both these methods, we propose an end-to-end pipeline that uses a face shape prior to estimate face pose, followed by stereo matching and finally a 3DMM to fill in the missing regions. Unlike SFM, which is highly under-constrained with two views, using a face shape prior makes face pose estimation much more stable and accurate. The proposed method is end-to-end topologically consistent, enabling a face pose refinement procedure to iteratively improve the face pose. The quantitative and qualitative results presented in this paper show that the proposed method improves MSE accuracy over existing state-of-the-art methods.