Adaptive Testing of Computer Vision Models

Irena Gao, Gabriel Ilharco, Scott Lundberg, Marco Tulio Ribeiro; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4003-4014


Vision models often fail systematically on groups of data that share common semantic characteristics (e.g., rare objects or unusual scenes), but identifying these failure modes is a challenge. We introduce AdaVision, an interactive process for testing vision models which helps users identify and fix coherent failure modes. Given a natural language description of a coherent group, AdaVision retrieves relevant images from LAION-5B with CLIP. The user then labels a small amount of data for model correctness, which is used in successive retrieval rounds to hill-climb towards high-error regions, refining the group definition. Once a group is saturated, AdaVision uses GPT-3 to suggest new group descriptions for the user to explore. We demonstrate the usefulness and generality of AdaVision in user studies, where users find major bugs in state-of-the-art classification, object detection, and image captioning models. These user-discovered groups have failure rates 2-3x higher than those surfaced by automatic error clustering methods. Finally, finetuning on examples found with AdaVision fixes the discovered bugs when evaluated on unseen examples, without degrading in-distribution accuracy, and while also improving performance on out-of-distribution datasets.

Related Material

[pdf] [supp] [arXiv]
@InProceedings{Gao_2023_ICCV, author = {Gao, Irena and Ilharco, Gabriel and Lundberg, Scott and Ribeiro, Marco Tulio}, title = {Adaptive Testing of Computer Vision Models}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2023}, pages = {4003-4014} }