3D Reconstruction Using Differential Imaging

Abstract

3D reconstruction is at the heart of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Because monocular 3D sensing is fundamentally ill-posed, many reconstruction techniques achieve high accuracy by using multiple captures to solve the inverse problem. Depending on the amount of change in these captures relative to the scale of the scene, we can broadly categorize imaging techniques into two groups: non-differential imaging and differential imaging. For example, a stereo system with a large baseline is non-differential, whereas one with a tiny baseline is differential.

Differential imaging offers a few advantages. On the hardware side, because of the tiny changes in measurements, differential imaging systems can be made compact and portable. There are commercially-available sensors at our disposal that already facilitate differential imaging, such as light field cameras and dual-pixel cameras that capture images of a scene under differentially-varying viewpoints. On the algorithm side, differential imaging makes it possible to locally linearize originally nonlinear phenomena so that inverse problems become easier to solve.

In this thesis, we leverage differential imaging to solve three challenging reconstruction problems. First, we introduce a method for non-line-of-sight (NLOS) imaging, an imaging scenario where the scene of interest is not directly visible to the camera. We apply differential imaging by densely scanning a visible surface using a transient imaging system. We then extract a geometric feature that we call the Fermat paths (defined as light paths that satisfy Fermat's principle) from each transient measurement of photons bouncing between the visible surface and the NLOS object. Using the collection of Fermat paths at all scan points, we apply tools from differential geometry to conduct differential analysis and reconstruct the surface of the NLOS objects.

Second, we introduce a method for reconstructing purely specular objects. Our setup illuminates the unknown object with a near-field point light source, and images it with a differentially translating camera. The interaction of light with specular surfaces, specular reflection, is a consequence of Fermat's principle. Therefore, our method for specular shape reconstruction also leverages the theory of Fermat paths. We examine both the geometric and radiometric information in these paths to show that it is possible to uniquely reconstruct the unknown specular object.

Third and last, we introduce a method for single-shot depth from defocus. This is a fundamentally ill-posed problem when using a conventional camera. To address this, we propose to use a commercially-available dual-pixel sensor, which emulates a stereo system with a differential baseline. We study the image formation model of a dual-pixel camera in the presence of defocus blur, and propose a method to simultaneously estimate the defocus map and the latent all-in-focus image from a single dual-pixel capture.

We hope this thesis will inspire the use of more differential imaging hardware systems and algorithms for 3D reconstruction. The techniques we introduce in this thesis, especially the theory of Fermat paths, will also apply to other domains, including wavefront sensing, acoustic and ultrasonic imaging, lensless imaging, and seismic imaging.