Learning Interpretable Queries for Explainable Image Classification with Information Pursuit

Stefan Kolek, Aditya Chattopadhyay, Kwan Ho Ryan Chan, Hector Andrade-Loarca, Gitta Kutyniok, René Vidal; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 3947-3956

Information Pursuit (IP) is a recently introduced learning framework to construct classifiers that are interpretable-by-design. Given a set of task-relevant and interpretable data queries, IP selects a small subset of the most informative queries and makes predictions based on the gathered query-answer pairs. However, a key limitation of IP is its dependency on task-relevant interpretable queries, which typically require considerable data annotation and curation efforts. While previous approaches have explored using general-purpose large language models to generate these query sets, they rely on prompt engineering heuristics and often yield suboptimal query sets, resulting in a performance gap between IP and non-interpretable black-box predictors. In this work, we propose parameterizing IP queries as a learnable dictionary defined in the latent space of vision-language models such as CLIP. We formulate an optimization objective to learn IP queries and propose an alternating optimization algorithm that shares appealing connections with classic sparse dictionary learning algorithms. Our learned dictionary outperforms baseline methods based on handcrafted or prompted dictionaries across several image classification benchmarks.