Cross-Modality Complementary Learning for Video-based Cloth-Changing Person Re-Identification

Vuong D. Nguyen, Pranav Mantini, Shishir K. Shah; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 88-107

Video-based Cloth-Changing Person Re-ID (VCCRe-ID) is a real-world Re-ID problem where individuals are observed in settings with a high likelihood of clothing changes between observations. To tackle this problem, capturing cloth-invariant modalities remains more effective than texture-based approaches. However, previous works extracted these modalities separately and directly leveraged the learned features for Re-ID, which is not effective since viewpoint changes and occlusion cause severe ambiguity in these modalities. To address this limitation, we propose a dual-branch framework that couples cloth-invariant modalities (i.e. shape and gait) with appearance by novelly exploiting the complementary relationship across them. In this work, we design a texture branch that enables body shape to complement the ambiguity in appearance caused by illumination variations or occlusions. Then texture and gait features are mutually learned at multiple levels, which helps to exchange beneficial information across branches for more discriminative person representations. We build a large-scale video-based cloth-changing dataset that contains the most cloth variations and is the first benchmark that mimics the real-world similar-clothing scenario. Extensive experiments show that our proposed framework outperforms state-of-the-art methods by a large margin. Code and dataset will be available at https://github.com/dustin-nguyen-qil/CCL-VCCReID.