ImageNet-BG: A Toolkit and Dataset for Evaluating Vision Model Robustness Against Background Variations

Łukasz Piekarek, Kamil Szyc; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 231-240

We present a toolkit that generates image datasets with diverse backgrounds and, using it, construct ImageNet-BG - a benchmark for probing the background robustness of modern vision models, from CNNs to ViTs. Using ImageNet-S, we extracted foreground objects and placed them on new real backgrounds from the SUN397 and Describable Textures Dataset and abstract backgrounds, including white, black, and Gaussian noise. Our contributions include the public release of the toolkit, the ImageNet-BG dataset, Human Annotations for Quality Assessment (HAQA) to assess background transferability, and a benchmark of popular pre-trained models. Our experiments reveal that even the latest models fall short of human-level performance, often experiencing significant drops in accuracy when backgrounds are altered. Notably, ConvNeXt-L shows the highest robustness but is still within a 6.4% decline for real backgrounds. Our per-class analysis indicates that many classes exhibit strong correlations with backgrounds, providing critical insights into model vulnerabilities.