Leveraging Pretrained Image Classifiers for Language-Based Segmentation

David Golub, Roberto Martin-Martin, Ahmed El-Kishky, Silvio Savarese; The IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 2010-2019

Abstract


Current semantic segmentation models cannot easily generalize to new object classes unseen during train time: they require additional annotated images and retraining. We propose a novel segmentation model that injects visual priors from pretrained image classifiers into semantic segmentation architectures, allowing them to segment out new target labels without retraining. As visual priors, we use the activations of pretrained image classifiers, which provide noisy indications of the spatial location of both the target object and distractor objects in the scene. We leverage language semantics to obtain these activations for a target label unseen by the classifier. Further experiments show that the visual priors obtained via language semantics for both relevant anddistracting objects are key to our performance

Related Material


[pdf]
[bibtex]
@InProceedings{Golub_2020_WACV,
author = {Golub, David and Martin-Martin, Roberto and El-Kishky, Ahmed and Savarese, Silvio},
title = {Leveraging Pretrained Image Classifiers for Language-Based Segmentation},
booktitle = {The IEEE Winter Conference on Applications of Computer Vision (WACV)},
month = {March},
year = {2020}
}