Measuring Hidden Bias Within Face Recognition via Racial Phenotypes

Seyma Yucer, Furkan Tektas, Noura Al Moubayed, Toby P. Breckon; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 995-1004

Recent work reports disparate performance for intersectional racial groups across face recognition tasks: face verification and identification. However, the definition of racial groups has a significant impact on the underlying findings of such racial bias analysis. Previous studies define these groups based on either demographic information (e.g. African, Asian etc.) or skin tone (e.g. lighter or darker skins). The use of such either sensitive or broad and loosely defined group definitions has disadvantages for both bias investigation and the design of subsequent counter-bias solutions. By contrast, this study introduces an alternative racial bias analysis methodology via the use of facial phenotype attributes for face recognition. We use the set of observable characteristics of an individual face where a race-related facial phenotype is hence specific to the human face and correlated to the racial profile of the subject. We propose categorical test cases to investigate the individual influence of those attributes on bias within face recognition tasks. We compare our phenotype-based grouping methodology with previous grouping strategies and show that phenotype-based groupings uncover hidden bias without exposing any potentially protected attributes. Furthermore, we contribute corresponding phenotype attribute category labels for face recognition tasks: RFW for face verification and VGGFace2 (test set) for face identification.