Seeing Out of the Box: End-to-End Pre-Training for Vision-Language Representation Learning

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 12976-12985

We study on joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based representations usually represent parts of an image, it is challenging for existing models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to ""See Out of tHe bOx"" that takes a full image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations, while enables 10 times faster inference than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics, and VD can be further updated on-the-fly during pre-training. We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR2 test-P split, 6.7% accuracy on SNLI-VE test split, respectively.