Lights, Camera, Render: Neural Fields for Structured Lighting

Abstract

3D scene reconstruction from 2D image supervision alone is an under-constrained problem. Recent neural rendering frameworks have made great strides in learning 3D scene representations to enable novel view synthesis, but they struggle to reconstruct geometry of low-texture regions or from sparse views. The prevalence of active depth sensors in common devices (e.g., iPhone, Kinect, RealSense) has stimulated the use of depth-supervised neural models to accurately disambiguate the scene’s geometry. However, the depth processed from these sensors can be prone to error, or even fail outright. Instead, a more principled approach is to explicitly model the raw structured light images themselves. In this work, we present an image formation model and optimization procedure that combines the advantages of neural radiance fields and structured light imaging. Our proposed approach enables the estimation of high-fidelity depth maps from sparse views, including for objects with complex material properties (e.g., partially-transparent surfaces). Additionally, the raw structured light images confer useful radiometric cues, which enable predicting surface normals and decomposing scene appearance in terms of a direct, indirect, and ambient component. We evaluate our framework quantitatively and qualitatively on a range of real and synthetic scenes, and decompose scenes into their constituent components for novel views.