Abstract:
With the popularity of Virtual Reality (VR), Augmented Reality (AR), and other 3D applications, developing methods that let everyday users capture and create their own 3D content has become increasingly essential.

Current 3D creation pipelines often require either tedious manual effort or specialized setups with densely captured views. Additionally, many resulting 3D models are incompatible with downstream applications due to inconsistent representations and baked-in lighting effects in textures.

My research addresses these challenges by leveraging data priors from other modalities, datasets, and large-scale diffusion models to reduce the burden on user input to casually captured photos, videos, and simple sketches. I will first show how depth priors can enable users to digitalize 3D scenes without dense data capture, and discuss how to enable interactive 3D editing and generation through 2D user inputs such as sketches. Moreover, I will also discuss how data and diffusion model priors can be utilized to generate relightable textures on meshes using text input, ensuring that generated 3D objects are functional in downstream applications, including game engines or digital production workflows.

Lastly, I will present ongoing work addressing challenges in large-scale 3D scene generation and incorporating manufacturing constraints into 3D content creation.

Thesis Committee Members:
Jun-Yan Zhu, Co-chair
Deva Ramanan, Co-chair
Shubham Tulsiani
Maneesh Agrawala, Stanford
Noah Snavely, Cornell Tech & Google