Quantifying the amount of visual information used by neural caption generators

Marc Tanti, Albert Gatt, Kenneth P. Camilleri; Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0-0

Abstract

This paper addresses the sensitivity of neural image caption generators to their visual input. A sensitivity analysis and omission analysis based on image foils is reported, showing that the extent to which image captioning architectures retain and are sensitive to visual information varies depending on the type of word being generated and the position in the caption as a whole. We motivate this work in the context of broader goals in the field to achieve more explainability in AI.

Related Material

[pdf]
[bibtex]
@InProceedings{Tanti_2018_ECCV_Workshops,
author = {Tanti, Marc and Gatt, Albert and Camilleri, Kenneth P.},
title = {Quantifying the amount of visual information used by neural caption generators},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops},
month = {September},
year = {2018}
}