Ego-Exo: Transferring Visual Representations From Third-Person to First-Person Videos

Yanghao Li, Tushar Nagarajan, Bo Xiong, Kristen Grauman; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6943-6953

Abstract


We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

Related Material


[pdf] [supp]
[bibtex]
@InProceedings{Li_2021_CVPR, author = {Li, Yanghao and Nagarajan, Tushar and Xiong, Bo and Grauman, Kristen}, title = {Ego-Exo: Transferring Visual Representations From Third-Person to First-Person Videos}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2021}, pages = {6943-6953} }