Unmasking Your Expression: Expression-Conditioned GAN for Masked Face Inpainting

Sridhar Sola, Darshan Gera; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 5908-5916

As face masks continue to be a part of our daily lives, the challenge of reconstructing occluded faces remains relevant. While several approaches have been proposed for removing masks from neutral facial images, few have explored the use of facial expressions as a dominant feature for reconstruction of expressive faces. To address this gap, we propose an expression-conditioned GAN (ECGAN) for reconstructing masked faces with a specified expression. Our approach leverages both the binary segmentation map of the mask and an expression label to generate high-quality images. To train our ECGAN in a supervised manner, we synthesize masked images using the RAFDB dataset to create non-masked-masked pairs of images for training. We evaluate of our approach on the RAFDB test set, demonstrating its effectiveness in generating realistic images that convincingly belong to the given expression class. This is further highlighted by comparing it to a baseline model and a state-of-the-art approach without expression-input. The code is available at https://github.com/SridharSola/ECGAN.