Barlow Constrained Optimization for Visual Question Answering

Abhishek Jha, Badri Patro, Luc Van Gool, Tinne Tuytelaars; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 1084-1093


Visual question answering is a vision-and-language multimodal task, that aims at predicting answers given samples from the question and image modalities. Most recentmethods focus on learning a good joint embedding space ofimages and questions, either by improving the interactionbetween these two modalities, or by making it a more discriminant space. However, how informative this joint space is, has not been well explored. In this paper, we propose a novel regularization for VQA models, Constrained Optimization using Barlow's theory (COB), that improves the information content of the joint space by minimizing the redundancy. It reduces the correlation between the learned feature components and thereby disentangles semantic concepts. Our model also aligns the joint space with the answer embedding space, where we consider the answer and image+question as two different 'views' of what in essence is the same semantic information. We propose a constrained optimization policy to balance the categorical and redundancy minimization forces. When built on the state-of-the-art GGE model, the resulting model improves VQA accuracy by 1.4% and 4% on the VQA-CP v2 and VQA v2 datasets respectively. The model also exhibits better interpretability. Code is made available:

Related Material

[pdf] [supp] [arXiv]
@InProceedings{Jha_2023_WACV, author = {Jha, Abhishek and Patro, Badri and Van Gool, Luc and Tuytelaars, Tinne}, title = {Barlow Constrained Optimization for Visual Question Answering}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2023}, pages = {1084-1093} }