Remote Pulse Estimation in the Presence of Face Masks

Jeremy Speth, Nathan Vance, Patrick Flynn, Kevin Bowyer, Adam Czajka; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022, pp. 2086-2095

Remote photoplethysmography (rPPG), a family of techniques for monitoring blood volume changes, may be especially useful for contactless health monitoring via face videos from consumer-grade cameras. The COVID-19 pandemic caused widespread use of protective face masks, which results in a domain shift from the typical region of interest. In this paper we show that augmenting unmasked face videos by adding patterned synthetic face masks forces the deep learning-based rPPG model to attend to the periocular and forehead regions, improving performance and closing the gap between masked and unmasked pulse estimation. This paper offers several novel contributions: (a) deep learning-based method designed for remote photoplethysmography in a presence of face masks, (b) new dataset acquired from 54 masked subjects with recordings of their face and ground-truth pulse waveforms, (c) data augmentation method to add a synthetic mask to a face video, and (d) evaluations of handcrafted algorithms and two 3D convolutional neural network-based architectures trained on videos of unmasked faces and with masks synthetically added.