Abstract:
We humans are able to understand 3D shapes of common daily objects and interact with them from a wide range of categories. We understand cups are usually cylinder-like and we can easily predict the shape of one particular cup, both in isolation or even when it is held by a human. We aim to build systems with similar ability, which can infer the 3D shape of isolated objects and even when humans are interacting with them and a desired system should be general for a wide range of common categories. However, current visual systems are typically limited to reconstructing a few categories due to lack of supervision or even requiring an approximate 3D template during inference. In the first part of my talk, we focus on reducing human supervision for single-view 3D object reconstruction. We propose a scalable method that can train from unstructured image collections and can be supervised only by segmentation masks that are predicted by an off-the-shelf system. By encouraging original-view consistency and novel view realism, our method is demonstrated to be scalable and effective on 50 common categories. The second part of my talk focuses on reconstructing hand-held objects. Prior works typically assume known 3D objects and reduce the problem to 6D pose estimation. We base on the observation that hand pose is predictive of the object it interacts with and propose an articulation-conditioned model to infer the object shape. We first use an off-the-shelf system to estimate hand pose and then infer the implicit signed distance field for the object in a normalized hand coordinate. We analyze the benefit of explicitly considering hand-object interactions and show remarkable improvement over baselines.

Research Qualifier Committee:
Abhinav Gupta (co-advisor)
Shubham Tulsiani (co-advisor)
Jun-Yan Zhu
Martin Li