Learning to Separate Object Sounds by Watching Unlabeled Video

Ruohan Gao, Rogerio S. Feris, Kristen Grauman; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018, pp. 2496-2499

Abstract


Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to study audio source separation in large-scale general "in the wild" videos. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising.

Related Material


[pdf] [arXiv]
[bibtex]
@InProceedings{Gao_2018_CVPR_Workshops,
author = {Gao, Ruohan and Feris, Rogerio S. and Grauman, Kristen},
title = {Learning to Separate Object Sounds by Watching Unlabeled Video},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2018}
}