Evolve: Enhancing Unsupervised Continual Learning With Multiple Experts

Xiaofan Yu, Tajana Rosing, Yunhui Guo; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 2366-2377

Recent years have seen significant progress in unsupervised continual learning methods. Despite their success in controlled settings, their practicality in real-world contexts remains uncertain. In this paper, we first empirically investigate existing self-supervised continual learning methods. We show that even with a replay buffer, existing methods cannot preserve the critical knowledge on videos with temporal-correlated input. Our insight is that the primary challenge of unsupervised continual learning stems from the unpredictable input and the absence of supervision as well as prior knowledge. Drawing inspiration from hybrid AI, we introduce EVOLVE, an innovative framework employing multiple pre-trained models in the cloud, as experts, to bolster existing self-supervised learning methods on local clients. EVOLVE harnesses expert guidance through a novel expert aggregation loss, calculated and returned from the cloud. It also dynamically assigns weights to experts based on their confidence and tailored prior knowledge, thereby offering adaptive supervision for new streaming data. We extensively validate EVOLVE across several real-world data streams with temporal correlation. The results convincingly demonstrate that EVOLVE surpasses the best state-of-the-art unsupervised continual learning method by 6.1-53.7% in top-1 linear evaluation accuracy across various data streams, affirming the efficacy of diverse expert guidance. The codebase is at https://github.com/Orienfish/Evolve.