End to End Lip Synchronization with a Temporal AutoEncoder

Yoav Shalev, Lior Wolf; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 341-350

Abstract


We study the problem of syncing the lip movement in a video with the audio stream. Our solution finds an optimal alignment using a dual-domain recurrent neural network that is trained on synthetic data we generate by dropping and duplicating video frames. Once the alignment is found, we modify the video in order to sync the two sources. Our method is shown to greatly outperform the literature methods on a variety of existing and new benchmarks. As an application, we demonstrate our ability to robustly align text-to-speech generated audio with an existing video stream. Our code is attached as supplementary.

Related Material


[pdf] [supp] [video]
[bibtex]
@InProceedings{Shalev_2020_WACV,
author = {Shalev, Yoav and Wolf, Lior},
title = {End to End Lip Synchronization with a Temporal AutoEncoder},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
month = {March},
year = {2020}
}