IMC: A Benchmark for Invariant Learning under Multiple Causes

Taero Kim, Seonggyun Lee, Joonseong Kang, Youngjun Choi, Wonsang Yun, Nicole Hee-Yeon Kim, Ziyu Chen, Lexing Xie, Kyungwoo Song; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2025, pp. 6535-6544

Deploying machine learning models in new environments requires robust algorithms that can handle distribution shifts. Invariant learning aims to achieve this by learning features that remain predictive across diverse environments. However, the previous benchmark dataset for invariant learning algorithms typically assumes that each class is determined by a single cause, leaving the study of scenarios where multiple causes contribute to class determination largely unexplored. To achieve a reliable performance in the real world, it should make reasonable decisions based on the available causes, even when some causes are unobserved. This necessitates learning and effectively utilizing all relevant causes. In this paper, we introduce a new benchmark designed to evaluate invariant learning in settings where multiple causes determine class labels. To enable systematic and quantitative evaluation, we construct new testbeds that allow per-cause assessment and analyze how well invariant learning algorithms capture diverse causes. Our study newly reveals that existing algorithms tend to focus disproportionately on specific causes, and we provide an in-depth analysis of this phenomenon. Our benchmark introduces six new datasets across computer vision and natural language processing (NLP) domains. Finally, we provide a simple but effective method for improving invariant learning algorithms to ensure a more comprehensive utilization of multiple causes.