What Holds Back Open-Vocabulary Segmentation?

Josip Šarić, Ivan Martinović, Matej Kristan, Siniša Šegvić; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 4256-4266

Abstract

Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the ground-truth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.

Related Material

[pdf]
[bibtex]
@InProceedings{Saric_2025_ICCV, author = {\v{S}ari\'c, Josip and Martinovi\'c, Ivan and Kristan, Matej and \v{S}egvi\'c, Sini\v{s}a}, title = {What Holds Back Open-Vocabulary Segmentation?}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops}, month = {October}, year = {2025}, pages = {4256-4266} }