Self-Supervised Learning of Visual Features Through Embedding Images Into Text Topic Spaces

Lluis Gomez, Yash Patel, Marcal Rusinol, Dimosthenis Karatzas, C. V. Jawahar; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4230-4239

Abstract


End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.

Related Material


[pdf] [arXiv] [poster]
[bibtex]
@InProceedings{Gomez_2017_CVPR,
author = {Gomez, Lluis and Patel, Yash and Rusinol, Marcal and Karatzas, Dimosthenis and Jawahar, C. V.},
title = {Self-Supervised Learning of Visual Features Through Embedding Images Into Text Topic Spaces},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {July},
year = {2017}
}