Improved Speech Reconstruction From Silent Video

Ariel Ephrat, Tavi Halperin, Shmuel Peleg; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 455-462


Speechreading is the task of inferring phonetic information from articulatory facial movements by observing them visually, and is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible and natural-sounding acoustic speech signal from silent video frames of a speaking person. We train our model on speakers from the GRID and TCD-TIMIT datasets, and evaluate the quality and intelligibility of reconstructed speech using common objective measurements. We show that speech predictions from the proposed model attain scores which indicate significantly improved quality over existing models. In addition, we show promising results towards reconstructing speech from an unconstrained dictionary.

Related Material

[pdf] [supp][arXiv]
author = {Ephrat, Ariel and Halperin, Tavi and Peleg, Shmuel},
title = {Improved Speech Reconstruction From Silent Video},
booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops},
month = {Oct},
year = {2017}