LIMITR: Leveraging Local Information for Medical Image-Text Representation

Gefen Dawidowicz, Elad Hirsch, Ayellet Tal; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 21165-21173

Abstract

Medical imaging analysis plays a critical role in the diagnosis and treatment of various medical conditions. This paper focuses on chest X-ray images and their corresponding radiological reports. It presents a new model that learns a joint X-ray image & report representation. The model is based on a novel alignment scheme between the visual data and the text, which takes into account both local and global information. Furthermore, the model integrates domain-specific information of two types -- lateral images and the consistent visual structure of chest images. Our representation is shown to benefit three types of retrieval tasks: text-image retrieval, class-based retrieval, and phrase-grounding.

Related Material

[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Dawidowicz_2023_ICCV, author = {Dawidowicz, Gefen and Hirsch, Elad and Tal, Ayellet}, title = {LIMITR: Leveraging Local Information for Medical Image-Text Representation}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2023}, pages = {21165-21173} }