Continual Learning of Personalized Generative Face Models with Experience Replay

Annie N. Wang, Luchao Qi, Roni Sengupta; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 2942-2951

We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances styles poses and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However for long-term deployment of these models with relatively smaller storage this simple random sampling-based replay technique also forgets past representations. Thus we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound. We introduce continual learning datasets for five celebrities along with the evaluation framework metrics and visualizations to examine this problem. See our project page for more details.