Towards Real-time Controllable Neural Face Avatars

Abstract

Neural Radiance Fields (NeRF) are compelling techniques for modeling dynamic 3D scenes from 2D image collections. These volumetric representations would be well suited for synthesizing novel facial expressions but for three problems. First, deformable NeRFs are object agnostic and model holistic movement of the scene: they can replay how the motion changes over time, but they cannot alter it in an interpretable way. Second, controllable volumetric representations typically require either time-consuming manual annotations or 3D supervision to provide semantic meaning to the scene. Third, classic NeRF-based methods rely on numerical integration which involves sampling hundreds of points across the ray, and evaluating the MLP at all of those locations, making them prohibitively slow for real-time applications. In this work, we propose a real-time controllable neural representation for face self-portraits, that solves all of these problems within a common framework, and it can rely on automated processing. We use automated facial action recognition (AFAR) to characterize facial expressions as a combination of action units (AU) and their intensities. AUs provide both the semantic locations and control labels for the system. We also extend the light field network, the re-formulations of radiance fields to oriented rays, to dynamic de-formations and hyperspace representations to accelerate the rendering speed. Our method outperforms competing methods for novel view and expression synthesis in terms of visual and anatomic fidelity of expressions, and also achieves an order of magnitude faster rendering speed than state-of-the-art methods.