3D Face Reconstruction from Monocular Video and its Applications In the Wild

Abstract

3D face reconstruction is a very popular field of computer vision due to its applications in social media, entertainment and health. However, ever since the introduction of 3D morphable models as facial priors, 3D face reconstruction has been dominated by reconstruction from single images due to its ease and proximity to 3D face alignment. Even so, single image reconstruction methods suffer from inconsistent reconstructions across time and view points. Hence a natural extension is to reconstruct 3D face shape from videos. Because of recent methods in single image reconstruction setting the standards for state-of-the-art reconstruction, we introduce a method to fuse single image reconstructions across multiple frames to create a more accurate reconstruction. Furthermore, the lack of structured video datasets that fully captures the face and provide 3D ground truth scans, led us to develop and release the 3DFAW-Video dataset and challenge. We also introduce a symmetric distance metric to benchmark reconstructions on the 3DFAW-Video dataset that is less affected by reconstruction density. Finally, we discuss the usage of 3D face reconstructions in two different applications to be deployed ’in-the-wild’. In particular, we illustrate applications in mask-sizing from a metric face 3D reconstruction, and present 3D face normalization as a technique to improve vision based non-contact heart rate estimation methods.