Seeing and Hearing Egocentric Actions: How Much Can We Learn?

Alejandro Cartas, Jordi Luque, Petia Radeva, Carlos Segura, Mariella Dimiccoli; The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 0-0

Abstract


Our interaction with the world is an inherently multimodal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial, and temporal streams. Experimental results on the EPIC-Kitchens dataset show that multimodal integration leads to better performance than unimodal approaches. In particular, we achieved a 5.18% improvement over the state of the art on verb classification.

Related Material


[pdf]
[bibtex]
@InProceedings{Cartas_2019_ICCV,
author = {Cartas, Alejandro and Luque, Jordi and Radeva, Petia and Segura, Carlos and Dimiccoli, Mariella},
title = {Seeing and Hearing Egocentric Actions: How Much Can We Learn?},
booktitle = {The IEEE International Conference on Computer Vision (ICCV) Workshops},
month = {Oct},
year = {2019}
}