Learning to Assemble Neural Module Tree Networks for Visual Grounding

Daqing Liu, Hanwang Zhang, Feng Wu, Zheng-Jun Zha; The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 4673-4682


Visual grounding, a task to ground (i.e., localize) natural language in images, essentially requires composite visual reasoning. However, existing methods over-simplify the composite nature of language into a monolithic sentence embedding or a coarse composition of subject-predicate-object triplet. In this paper, we propose to ground natural language in an intuitive, explainable, and composite fashion as it should be. In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed. NMTree disentangles the visual grounding from the composite reasoning, allowing the former to only focus on primitive and easy-to-generalize patterns. To reduce the impact of parsing errors, we train the modules and their assembly end-to-end by using the Gumbel-Softmax approximation and its straight-through gradient estimator, accounting for the discrete nature of module assembly. Overall, the proposed NMTree consistently outperforms the state-of-the-arts on several benchmarks. Qualitative results show explainable grounding score calculation in great detail.

Related Material

[pdf] [supp]
author = {Liu, Daqing and Zhang, Hanwang and Wu, Feng and Zha, Zheng-Jun},
title = {Learning to Assemble Neural Module Tree Networks for Visual Grounding},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}