Are Exemplar-Based Class Incremental Learning Models Victim of Black-Box Poison Attacks?

Neeresh Kumar Perla, Md. Iqbal Hossain, Afia Sajeeda, Ming Shao; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 6785-6794

Class Incremental Learning (CIL) models are designed to continuously learn new classes without forgetting previously learned ones often relying on an exemplar set to retain a portion of knowledge from previously learned classes. However their vulnerability to adversarial attacks under novel and unexplored conditions remains unstudied. In this work we are the first to evaluate the robustness of exemplar-based CIL models using a non-overlapping dataset where the dataset is independent of the training and test sets of the target model. We propose and implement a novel black-box attack framework targeting the exemplar set of class incremental learning models using zero-overlapping data. Specifically we focus on scenarios where the target model provides only hard-label predictions without interactive access. Our experimental evaluation covers a range of exemplar-based incremental learning algorithms different surrogate models and black-box attack options. Our findings reveal significant vulnerabilities in exemplar-based CIL models to poisoning-based attacks using a non-overlapping dataset.