Event Location: NSH 1305

Abstract: This thesis provides a system that allows users to interact with objects and human subjects in photographs in three dimensions. Current photo-editing tools provide a diverse array of operations in photographs. However, they are limited to the 2D plane of the image. The key contribution of this thesis is to leverage 3D models from public repositories to obtain a three-dimensional representation of objects in photographs for perceptually plausible seamless manipulation of photographed objects. The approach described in this thesis addresses the mismatch between the geometry and appearance of 3D models from photographed instances.

To correct the mismatch between the geometry of the 3D model and the photographed object, the thesis discusses an automatic model alignment technique that uses information from local patches distributed within the object to predict the global transformation and local deformation of the 3D model to the photographed object. We also present a manual geometry adjustment tool that allows users to perform final corrections while imposing smoothness and symmetry constraints. Given the matched geometry, we use the visible pixels to estimate unknown illumination, correct the mismatched appearance, and complete the appearance in hidden areas using symmetries. Our approach provides users with intuitive three-dimensional control over objects to perform manipulations impossible in current photo-editing tools.

Currently, our approach allows users to manipulate inanimate objects in photographs. In this thesis, we propose to provide three-dimensional control over photographed human subjects. Representing human subjects with 3D models is challenging as a single 3D model may not capture all photographed people due to differences in geometric structure and appearance introduced by variation in clothing appearance and style, body shape, and articulation. In the remainder of this thesis, we propose to adapt our automatic model alignment, illumination estimation, and appearance completion approaches to provide three-dimensional manipulation of human subjects in photographs.

Committee:Yaser Sheikh, Chair

Nancy Pollard

Kayvon Fatahalian

Alexei Efros, University of California, Berkeley

David Forsyth, University of Illinois at Urbana-Champaign