One VLM to Keep it Learning: Generation and Balancing for Data-Free Continual Visual Question Answering

Deepayan Das, Davide Talon, Massimiliano Mancini, Yiming Wang, Elisa Ricci; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 5635-5645

Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting rehearsal strategy uses the data of past tasks upon learning new task. However such strategy incurs the need of storing past data which might not be feasible due to hardware constraints or privacy concerns. In this work we propose the first data-free method that leverages the language generation capability of a VLM instead of relying on external models to produce pseudo-rehearsal data for addressing continual VQA. Our proposal named as GaB generates pseudo-rehearsal data by posing previous task questions on new task data. Yet despite being effective the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks i.e. VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks while being on-par with methods with access to the past data. Code will be made public soon.