Given 3 views of a held-out test subject from our dataset, we show high-quality novel view synthesis. Note the 3D consistent rendering of details such as hair strands, eye-lashes, and view-dependent effects for example on the forehead.
Please note that due to the file size limit (100MB), we can only show limited high-resolution video results.
We show novel view synthesis results at 2K resolution using only two input views of subjects from the FaceScape dataset. Note that our prior model was trained on a different dataset—these results represent the out-of-distribution setting.
We demonstrate the generalization capability of our method to in-the-wild mobile camera captures. With just 2 input views, our method is able to genenerate highly consistent and photorealistic free-view renders of a subject. Our method not only reconstructs coherent geometry, it also learns to interpolate view-dependent specularities, such as on the hair and skin.
Synthesis from single view naturally suffers from bas-relief ambiguity making it much more challenging, but our prior model enables plausible results. We show results on our studio captures and one in-the-wild example.
Alignment of faces in our dataset allows us to learn a continuous latent space, where the embeddings of training identities can be interpolated to achieve plausible intermediate identities. Note that we do not train our model in an adversarial manner but only with reconstruction losses.
We show the effect of using regularization during model fitting. Note the colour distortions in the absence of view regularization and the fuzzy surfaces without the normal regularization.
The model finetuning has to be initialised correctly in order to avoid artifacts like floaters or duplicate body parts, e.g., ears.
Our method allows novel view synthesis from extremely sparse view, even from a single image. The examples below show the rendering quality improves when a variable number of views are available.
We train the prior model on fewer number of identities and lower resolution. The results show that a more diverse prior model performs better while a lower resolution prior model might not necessarily be required.
We visualise the image-space geometry estimated by our method. Note the 3D consistent depth and normals. The normals in the hair have a grey appearance due to transparent density.
While our method achieves state-of-the-art results in high-resolution synthesis of faces, it struggles with strong expressions and large accessories. This is due to limited representation in our training dataset. The training dataset only contains neutral faces and none of the subjects was wearing voluminous clothing like jackets. This limitation could potentially be mitigated by training a more diverse prior model that includes those modalities.
