Copy or Not? Reference-Based Face Image Restoration with Fine Details

Min Jin Chong, Dejia Xu, Yi Zhang, Zhangyang Wang, David Forsyth, Gurunandan Krishnan, Yicheng Wu, Jian Wang; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 9642-9651

Reference-guided face restoration can have better identity preservation than non-reference-based methods. However existing methods can (a) easily produce artifacts possibly attributable to inefficient facial priors and (b) do not well preserve fine-grained facial details crucial for identity such as freckles tattoos and scars. In this work we propose solutions for these problems. (1) We incorporate a stronger facial prior generative facial prior (GFP) for reference-based face image restoration. (2) We identify an ambiguity and point out that traditional loss prevents the network from heavily copying facial features from the reference. To address this we set a new goal and come up with a new loss to realize the new goal. More specifically when the ground truth and reference are different (e.g. differences in wrinkles makeup facial hair etc.) which one should the output look like? As a simple example ground truth does not have a mole while reference has one. Traditional loss chose the ground truth which seems natural but then the network also learns to ignore reference's facial features; during testing the network often hesitates. Our new goal is to copy features from the reference as much as possible while maintaining semantic consistency with the degraded input. We propose to use spatial minimum loss and cycle consistency loss to realize the new goal and make the network copy features without hesitation. Using only a single reference image our proposed method is able to restore highly degraded images while accurately capturing fine-grained facial details. To our knowledge we are the first face restoration framework that is able to restore faces at this granularity. Code and data are available at https://github.com/RefineFIR/RefineFIR.