Abstract:
3D reconstruction has been at the core of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Because monocular 3D sensing is fundamentally ill-posed, many techniques aiming for accurate reconstruction use multiple captures to solve the inverse problem. Depending on the amount of change in these captures relative to the scale of the scene, imaging methods can be categorized into two groups: non-differential imaging and differential imaging. For example, a stereo system with a large baseline is considered non-differential, while one with a tiny baseline is considered differential.

Differential imaging offers a few advantages over its non-differential counterparts. On the hardware side, because of the tiny changes in measurements, differential imaging systems can be made compact and portable. There are commercially-available sensors at our disposal that already facilitate differential imaging, such as light field cameras that capture images of a scene under slightly varying viewpoints. On the algorithm side, differential imaging makes it possible to locally linearize originally nonlinear phenomena so that inverse problems become easier to solve.

This thesis leverages differential imaging to solve three challenging reconstruction problems. First, we propose a novel method for non-line-of-sight (NLOS) imaging, a challenging scenario where the scene of interest is not directly visible. We apply differential imaging by densely scanning a visible surface using a transient imaging system. We then extract a geometric feature that we call the Fermat paths (defined as light paths that satisfy Fermat’s principle) from each transient measurement of photons bouncing between the visible surface and the NLOS object. Using the collection of Fermat paths at all scan points, we apply well-established tools in differential geometry to conduct differential analysis and reconstruct the surface of NLOS objects. Next, we consider the problem of reconstructing purely specular mirror-like objects illuminated by a near-field point light source and imaged by a differentially translating camera. The light interaction of mirror surfaces follows the law of specular reflection, which is derived from Fermat’s principle. Therefore, our adopted method for specular shape reconstruction is also based on the theory of Fermat paths. We further examine the radiometric information of these specularities to eliminate the ambiguity in the reconstruction using only the geometric information of Fermat paths. Finally, we address the problem of single-shot depth from defocus, which is fundamentally ill-posed using a conventional camera. We propose to use a commercially-available dual-pixel (DP) sensor, which emulates a stereo system with a differential baseline. We study the image formation model of a DP camera in the presence of defocus blur, and propose a method to simultaneously estimate the defocus map and the latent all-in-focus image from a single DP capture.

We hope this thesis will inspire the use of more differential imaging systems and algorithms for 3D reconstruction. The techniques developed in this thesis, especially the theory of Fermat paths, will also apply to other domains, including wavefront sensing, acoustic and ultrasound imaging, lensless imaging, and seismic imaging.

Thesis Committee Members:
Ioannis Gkioulekas, Co-chair
Srinivasa G. Narasimhan, Co-chair
Keenan Crane
Gordon Wetzstein, Stanford

More Information