Who Are You Referring To? Coreference Resolution In Image Narrations

Arushi Goel, Basura Fernando, Frank Keller, Hakan Bilen; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 15247-15258

Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coreference chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.