Abstract:  Humans have extraordinary capabilities of comprehending and reasoning about our 3D visual world. One particular reason is that when looking at an object or a scene, not only can we see the visible surface, but we can also hallucinate the invisible parts – the amodal structure, appearance, affordance, etc. We have accumulated thousands of hours of observations about our world that help us form a mental model from which we can extrapolate.

In this talk, I will argue that equipping machines with mental models is the key towards solving various 3D tasks in computer vision and robotics. I will start by discussing 3D reconstruction from sparse views, where the camera poses are unknown and the images have little or no overlap. While existing approaches struggle in this setting, I will show that by distilling prior knowledge about objects into the algorithms, we can significantly expand their applicable domains and unleash the potential of multiple downstream tasks in extreme-view setups. Then, I will present our latest effort towards generating a realistic and consistent 3D world from a single RGBD image. The task is highly challenging since the output often drifts apart and there is usually no loop-closure, resulting in incoherent scenes. I will show that integrating a “map” into the hallucination process can drastically alleviate the issues. Finally, I will showcase how we exploit mental models to enable more efficient robot manipulation and build sensor simulation systems for self-driving.

Bio: Wei-Chiu Ma is a final-year Ph.D. candidate at MIT, working with Antonio Torralba and Raquel Urtasun. His research lies in the intersection of computer vision, robotics, and graphics, with a focus on in-the-wild 3D modeling and simulation and their applications to self-driving vehicles. Wei-Chiu is a recipient of the Siebel Scholarship. His work has been covered by media outlets such as WIRED, DeepLearning.AI, MIT News, etc. Previously, Wei-Chiu was a Sr. Research Scientist at Uber ATG R&D. He received his M.S. in Robotics from CMU where he was advised by Kris Kitani and B.S. in EE from National Taiwan University.

Homepage:   http://people.csail.mit.edu/weichium/

Sponsored in part by:   Facebook Reality Labs Pittsburgh