Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

Dogucan Yaman, Fevziye Irem Eyiokur, Leonard Bärmann, Seymanur Akti, Hazım Kemal Ekenel, Alexander Waibel; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 6003-6013

Abstract

In the task of talking face generation the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality as well as robustly evaluating such synchronization. To tackle these problems we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover leveraging AV-HuBERT's features we introduce three novel lip synchronization evaluation metrics aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results along with a detailed ablation study demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.

Related Material

[pdf] [supp]
[bibtex]
@InProceedings{Yaman_2024_CVPR, author = {Yaman, Dogucan and Eyiokur, Fevziye Irem and B\"armann, Leonard and Akti, Seymanur and Ekenel, Haz{\i}m Kemal and Waibel, Alexander}, title = {Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, month = {June}, year = {2024}, pages = {6003-6013} }