Exploring Image Classification Robustness and Interpretability with Right for the Right Reasons Data Augmentation

Flávio Arthur Oliveira Santos, Cleber Zanchettin; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 4147-4156

Right for the right reasons (RRR) methods have been proposed to mitigate the issues of shortcut learning in deep learning models. During training, these methods guide the models to learn patterns from signal information while ignoring noisy features. This work investigates the robustness of image classification models to background sensitivity, referring to a model's capability to accurately classify an image without leveraging the shortcut learning between the image background and the assigned input label. We propose a new approach, the Right for the Right Reasons Data Augmentation (RRDA). This approach augments the image foreground context with the context extracted from different images, thereby stimulating the model to focus on signal features rather than the context. Our experiments demonstrate that RRDA can significantly improve the robustness of image classification models, outperforming other RRR methods, such as GradMask and ActDiff. We also evaluate the impact of architectural choice on robustness, showing that ViT is more robust than ResNet in handling background sensitivity. Finally, we perform an interpretability analysis to understand how models assign importance to signal and context features during the inference process. This involves computing the signal-to-noise ratio as the importance of the signal divided by the importance of the context. Contrary to our expectations, our findings suggest that a high signal-to-noise ratio does not necessarily imply robustness. However, they indicate that applying RRDA can help the models learn to focus on signal features, leading to more interpretable and robust models.