Abstract:
In this talk we will discuss how to infer 3D from images captured by a single camera, without assuming the target scenes / objects being static. The non-static setting makes our problem ill-posed and challenging to solve, but is vital in practical applications where target-of-interest is non-static. To solve ill-posed problems, the current trend in the field is to learn inference models e.g. neural networks on datasets with labeled groundtruth. Instead, we attempt a data-less approach without requiring datasets with 3D annotations. This poor man’s approach is beneficial to tasks which lack well annotated datasets.

This talk will consist of two parts.

(i) We first introduce our series of works on non-rigid structure from motion (NR-SfM) and its application to learn 3D landmark detectors with only 2D landmark annotations. We propose techniques to make the NR-SfM algorithm scalable to large datasets and robust to missing data. We also propose a new loss to let the 3D landmark detector learn more efficiently from the NR-SfM teacher.

(ii) We then present works on reconstructing dense dynamic scenes. Dense reconstruction is challenging for NR-SfM algorithms mainly due to the difficulty in getting reliable long-term correspondences for every pixel. On the other hand, being able to reconstruct every pixel of the scene is necessary for applications like novel view synthesis. Therefore, we investigate solutions without the need for long-term correspondences. We explore an analysis-by-synthesis approach, where we leverage recent advances in differentiable neural rendering and represent dynamic scenes using deformable neural radiance fields (D-NeRF). We present a new method for D-NeRFs that can directly use optical flow as supervision. We overcome the major challenge with respect to the computational inefficiency of enforcing the flow constraints to the deformation field used by D-NeRFs. We present results on novel view synthesis with rapid object motion, and demonstrate significant improvements over baselines without flow supervision.

Thesis Committee Members:
Simon Lucey, Co-chair
Laszlo A Jeni, Co-chair
Fernando De La Torre
Katerina Fragkiadaki
Hongdong Li , Australian National University

More Information