Causal Feature Alignment: Learning To Ignore Spurious Background Features

Rahul Venkataramani, Parag Dutta, Vikram Melapudi, Ambedkar Dukkipati; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 4666-4674

Deep neural networks are susceptible to spurious features strongly correlating with the target. This phenomenon leads to sub-optimal performance during real-world deployment where the spurious correlations do not exist, leading to deployment challenges in safety-critical environments like healthcare, autonomous navigation etc. While spurious features can correlate with causal features in myriad ways, we propose a solution for a common manifestation in computer vision where the background corresponds to a spurious feature. In contrast to previous works, we do not require apriori knowledge of different sub-groups in the data induced by the presence/absence of spurious features and the corresponding access to samples from these sub-groups. Our proposed method, Causal Feature Alignment (CFA), utilizes segmentation of foreground (a proxy for the causal component) on a small subset of training examples to align the representations of the original images to match words from only causal elements. We first demonstrate the validity of the proposed method on semi-synthetic data. Subsequently, we obtain state-of-the-art results on worst-group accuracy (93%) on the benchmark dataset of Waterbirds using CFA. Furthermore, we demonstrate significant gains of 6% on the Backgrounds Challenge. Finally, we show that utilizing the recently released foundational methods can alleviate the requirement of dense segmentation and can be substituted with weaker modes of human input like bounding boxes, clicks etc., without any performance loss compared to the original CFA.