The Universal Face Encoder: Learning Disentangled Representations Across Different Attributes

Sandipan Banerjee, Ajjen Joshi, Jay Turcot; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 1071-1080

Models that can learn orthogonal representations for different facial attributes (eg. pose, lighting, identity, expressions) have proven to be beneficial for both discriminative and generative tasks. In this work, we propose the universal facial encoder (UFE) that can simultaneously encode different facial attributes as disentangled features from a single face image. We propose a variety of qualitative and quantitative metrics to evaluate feature orthogonality of the UFE and demonstrate superior disentanglement compared to traditional single-attribute encoding. We also show that these features can then be used to train lightweight prediction heads for multiple downstream classification tasks. Moreover, coupling the UFE with a style-based decoder enables hallucination of new face images composed of attributes taken from different samples. As experimentally demonstrated, the UFE allows us to pick and choose these attributes from label-disjoint datasets. A catalog of such synthetic composites can be used as supplemental training data or simply as stock photos.