Abstract:
In this proposal, we study how to infer 3D from images captured by a single camera, without assuming the target scenes / objects being static. The non-static setting makes our problem ill-posed and challenging to solve, but is vital in practical applications where target-of-interest is non-static. To solve ill-posed problems, the current trend in the field is to learn inference models \eg neural networks on datasets with labeled groundtruth. Instead, we attempt a data-less approach without requiring datasets with 3D annotations. This poor man’s approach is beneficial to tasks which lack well annotated datasets.

Our works are grouped into two topics.

(i) We first introduce our series of works on how to learn 3D landmark detectors with only 2D landmark annotations. Our general framework is a two stage approach — we design a novel non-rigid structure from motion (NRSfM) module to reconstruct shape and camera poses from input 2D landmarks, and then these are used to teach a neural network to detect 3D landmarks from image inputs. We propose techniques to make the NRSfM module scalable to large datasets and robust to missing data. We also propose a new loss to let the 3D landmark detector learn more efficiently from the NRSfM module.

(ii) We then present works on reconstructing dynamic scenes from videos. As a preliminary exploration, we take a data-driven approach, by collecting stereoscopic videos from Internet and train a depth estimation network with two frames as input. Despite the simplicity of the approach, it lacks geometric reasoning and consequently limited in its generalizability. In our proposed work, we explore an optimization-based approach. We leverage recent advances in differentiable neural rendering and combine it to trajectory-based NRSfM. We show some preliminary result and discuss the current challenges we have and some potential solutions.

Thesis Committee Members:
Simon Lucey, Co-chair
Laszlo A. Jeni, Co-chair
Fernando De La Torre
Katerina Fragkiadaki
Hongdong Li, Australian National University

More Information