Abstract:

Reconstructing scenes to synthesize novel views is a long standing problem in Computer Vision and Graphics. Recently, implicit scene representations have shown novel view synthesis results of unprecedented quality, like the ones of Neural Radiance Fields (NeRF), which use the weights of a multi-layer perceptron to model the volumetric density and color of a scene. In this talk, I will present two works that extend such representations to handle real-world data of scenes and deformable objects that one may capture with a smartphone. First, I will introduce a method to reconstruct landmarks using uncontrolled internet photo collections. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena, such as variable illumination or transient occluders. Our model – NeRF-W – is a series of extensions to NeRF to address these issues, thereby allowing for accurate reconstructions from in-the-wild photo collections. Next, I will present a method capable of photorealistically reconstructing a non-rigidly deforming scene using photos/videos captured casually from mobile phones. Our approach augments NeRF by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical template NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine and elastic regularizations that allow for more robust optimization. Our model – D-NeRF – can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub “nerfies”.

BIO:

Ricardo Martin-Brualla is a researcher at Google Seattle working on the future of communication. He completed his PhD in 2016 at the University of Washington advised by Steve Seitz. His research interests lie at the intersection of 3D reconstruction and neural rendering.

Homepage : ricardomartinbrualla.com

Sponsored in part by:   Facebook Reality Labs Pittsburgh