Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval With Partial Query

Guanyu Cai, Jun Zhang, Xinyang Jiang, Yifei Gong, Lianghua He, Fufu Yu, Pai Peng, Xiaowei Guo, Feiyue Huang, Xing Sun; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 1835-1844


Text-based image retrieval has seen considerable progress in recent years. However, the performance of existing methods suffers in real life since the user is likely to provide an incomplete description of an image, which often leads to results filled with false positives that fit the incomplete description. In this work, we introduce the partial-query problem and extensively analyze its influence on text-based image retrieval. Previous interactive methods tackle the problem by passively receiving users' feedback to supplement the incomplete query iteratively, which is time-consuming and requires heavy user effort. Instead, we propose a novel retrieval framework that conducts the interactive process in an Ask-and-Confirm fashion, where AI actively searches for discriminative details missing in the current query, and users only need to confirm AI's proposal. Specifically, we propose an object-based interaction to make the interactive retrieval more user-friendly and present a reinforcement-learning-based policy to search for discriminative objects. Furthermore, since fully-supervised training is often infeasible due to the difficulty of obtaining human-machine dialog data, we present a weakly-supervised training strategy that needs no human-annotated dialogs other than a text-image dataset. Experiments show that our framework significantly improves the performance of text-based image retrieval. Code is available at

Related Material

[pdf] [supp]
@InProceedings{Cai_2021_ICCV, author = {Cai, Guanyu and Zhang, Jun and Jiang, Xinyang and Gong, Yifei and He, Lianghua and Yu, Fufu and Peng, Pai and Guo, Xiaowei and Huang, Feiyue and Sun, Xing}, title = {Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval With Partial Query}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {1835-1844} }