Multimodal Data Augmentation for Visual-Infrared Person ReID With Corrupted Data

Arthur Josi, Mahdi Alehdaghi, Rafael M. O. Cruz, Eric Granger; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2023, pp. 32-41

The re-identification (ReID) of individuals over a complex|network of cameras is a challenging task, especially|under real-world surveillance conditions. Several deep|learning models have been proposed for visible-infrared (VI)|person ReID to recognize individuals from images captured|using RGB and IR cameras. However, performance|may decline considerably if RGB and IR images captured at|test time are corrupted (e.g., noise, blur, and weather conditions).|Although various data augmentation (DA) methods|have been explored to improve the generalization capacity,|these are not adapted for V-I person ReID. In this|paper, a specialized DA strategy is proposed to address|this multimodal setting. Given both the V and I modalities,|this strategy allows to diminish the impact of corruption|on the accuracy of deep person ReID models. Corruption|may be modality-specific, and an additional modality|often provides complementary information. Our multimodal|DA strategy is designed specifically to encourage|modality collaboration and reinforce generalization capability.|For instance, punctual masking of modalities forces|the model to select the informative modality. Local DA is|also explored for advanced selection of features within and|among modalities. The impact of training baseline fusion|models for V-I person ReID using the proposed multimodal|DA strategy is assessed on corrupted versions of the SYSUMM01,|RegDB, and ThermalWORLD datasets in terms of|complexity and efficiency. Results indicate that using our|strategy provides V-I ReID models the ability to exploit both|shared and individual modality knowledge so they can outperform|models trained with no or unimodal DA. GitHub|code: https://github.com/art2611/ML-MDA.