Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces

Chirag Raman, Charlie Hewitt, Erroll Wood, Tadas BaltruĊĦaitis; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 3515-3525

Abstract


Recent advances in synthesizing realistic faces have shown that synthetic training data can replace real data for various face-related computer vision tasks. A question arises: how important is realism? Is the pursuit of photorealism excessive? In this work, we show otherwise. We boost the realism of our synthetic faces by introducing dynamic skin wrinkles in response to facial expressions, and observe significant performance improvements in downstream computer vision tasks. Previous approaches for producing such wrinkles either required prohibitive artist effort to scale across identities and expressions, or were not capable of reconstructing high-frequency skin details with sufficient fidelity. Our key contribution is an approach that produces realistic wrinkles across a large and diverse population of digital humans. Concretely, we formalize the concept of mesh-tension and use it to aggregate possible wrinkles from high-quality expression scans into albedo and displacement texture maps. At synthesis, we use these maps to produce wrinkles even for expressions not represented in the source scans. Additionally, to provide a more nuanced indicator of model performance under deformations resulting from compressed expressions, we introduce the 300W-winks evaluation subset and the Pexels dataset of closed eyes and winks.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Raman_2023_WACV, author = {Raman, Chirag and Hewitt, Charlie and Wood, Erroll and Baltru\v{s}aitis, Tadas}, title = {Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2023}, pages = {3515-3525} }