Asymmetric Image Retrieval With Cross Model Compatible Ensembles

Alon Shoshan, Ori Linial, Nadav Bhonker, Elad Hirsch, Lior Zamir, Igor Kviatkovsky, Gérard Medioni; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 1-11


The asymmetrical retrieval setting is a well suited solution for resource constrained applications such as face recognition and image retrieval. In this setting, a large model is used for indexing the gallery while a lightweight model is used for querying. The key principle in such systems is ensuring that both models share the same embedding space. Most methods in this domain are based on knowledge distillation. While useful, they suffer from several drawbacks: they are upper-bounded by the performance of the single best model found and cannot be extended to use an ensemble of models in a straightforward manner. In this paper we present an approach that does not rely on knowledge distillation, rather it utilizes embedding transformation models. This allows the use of N independently trained and diverse gallery models (e.g., trained on different datasets or having a different architecture) and a single query model. As a result, we improve the overall accuracy beyond that of any single model while maintaining a low computational budget for querying. Additionally, we propose a gallery image rejection method that utilizes the diversity between multiple transformed embeddings to estimate the uncertainty of gallery images.

Related Material

[pdf] [supp] [arXiv]
@InProceedings{Shoshan_2024_WACV, author = {Shoshan, Alon and Linial, Ori and Bhonker, Nadav and Hirsch, Elad and Zamir, Lior and Kviatkovsky, Igor and Medioni, G\'erard}, title = {Asymmetric Image Retrieval With Cross Model Compatible Ensembles}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2024}, pages = {1-11} }