Face-swapping based Data Augmentation for ID Document and Selfie Face Verification

Elmokhtar Mohamed Moussa, Ioannis Sarridis, Emmanouil Krasanakis, Nathan Ramoly, Symeon Papadopoulos, Ahmad-Montaser Awal, Lara Younes; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 1421-1428

In this work we focus on identity verification by matching selfies to identity (ID) documents depicting faces. When applied to this task state-of-the-art deep face verification models tend to underperform due to lack of specific visual features in the documents in public datasets and suffer from skin tone biases due to representation imbalances of skin tone groups in training data. To address both issues we introduce a framework that generates ID documents containing synthetic face images to augment training datasets. Specifically we transform selfies into document-style images by combining style banks of document templates and face-swapping generative models. Experiments on a public and a proprietary real-world dataset reveal that fine-tuning state-of-the-art face verification models with the proposed methodology yields 7.96% improvement in verification accuracy while requiring only 25% of the original training data. Furthermore improvements occur across all skin tone groups including darker skin tones; though it is notoriously hard to perform accurate verification for this group we achieve more than 50% relative reduction in false acceptance and false rejection rate gaps w.r.t. lighter skin tones.