All in One Framework for Multimodal Re-identification in the Wild

He Li, Mang Ye, Ming Zhang, Bo Du; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 17459-17469

In Re-identification (ReID) recent advancements yield noteworthy progress in both unimodal and cross-modal retrieval tasks. However the challenge persists in developing a unified framework that could effectively handle varying multimodal data including RGB infrared sketches and textual information. Additionally the emergence of large-scale models shows promising performance in various vision tasks but the foundation model in ReID is still blank. In response to these challenges a novel multimodal learning paradigm for ReID is introduced referred to as All-in-One (AIO) which harnesses a frozen pre-trained big model as an encoder enabling effective multimodal retrieval without additional fine-tuning. The diverse multimodal data in AIO are seamlessly tokenized into a unified space allowing the modality-shared frozen encoder to extract identity-consistent features comprehensively across all modalities. Furthermore a meticulously crafted ensemble of cross-modality heads is designed to guide the learning trajectory. AIO is the first framework to perform all-in-one ReID encompassing four commonly used modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts showcasing exceptional performance in zero-shot and domain generalization scenarios. Code will be available at: https://github.com/lihe404/AIO.