Interactive Visual Feature Search

Devon Ulrich, Ruth Fong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8188-8193

Many visualization techniques have been created to explain the behavior of computer vision models but they largely consist of static diagrams that convey limited information. Interactive visualizations allow users to more easily interpret a model's behavior but most are not reusable for new models. We introduce Visual Feature Search a novel interactive visualization that is adaptable to most modern vision models and can easily be incorporated into a researcher's workflow. Our tool allows a user to highlight an image region and search for images from a given dataset with the most similar model features. We demonstrate how our tool elucidates different aspects of model behavior by performing experiments on a range of applications such as in medical imaging and wildlife classification. Our tool is open source and can be used by others to interpret their own models.