Cross-Modal Deep Variational Hashing

Venice Erin Liong, Jiwen Lu, Yap-Peng Tan, Jie Zhou; The IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4077-4085


In this paper, we propose a cross-modal deep variational hashing (CMDVH) method to learn compact binary codes for cross-modality multimedia retrieval. Unlike most existing cross-modal hashing methods which learn a single pair of projections to map each example into a binary vector, we design a deep fusion neural network to learn non-linear transformations from image-text input pairs, such that a unified binary code is achieved in a discrete and discriminative manner using a classification-based hinge-loss criterion. We then design modality-specific neural networks in a probabilistic manner such that we model a latent variable to be close as possible from the inferred binary codes, at the same time approximated by a posterior distribution regularized by a known prior, which is suitable for out-of-sample extension. Experimental results on three benchmark datasets show the efficacy of the proposed approach.

Related Material

author = {Erin Liong, Venice and Lu, Jiwen and Tan, Yap-Peng and Zhou, Jie},
title = {Cross-Modal Deep Variational Hashing},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {Oct},
year = {2017}