Title: Generating 3D Human Animations from Single Monocular Images

Abstract:

Endowing AI systems with the ability to formulate a three-dimensional understanding of human appearance from a single RGB image is an important component technology for applications such as person re-identification, biometrics, virtual reality and augmented reality. However, jointly inferring the texture map and 3D human model from a single image is challenging because: (1) scale ambiguity, clothing and occlusion make it hard to obtain the 3D model of a person from a single monocular image; and (2) a complete texture map needs to be inferred for each part of the body, such as skin tone, clothing and facial details, even if parts are occluded or are not visible in the image. With these challenges in mind, we introduce a methodology to generate the complete texture of a 3D model of a human from a single image. In particular, we use a model-based approach for estimating the body-shape and pose of the 3D mesh model and concurrently utilize it towards a data-driven approach for estimating the UV texture map of the body (e.g., shirt, pants, skin tone, face). 

We also show how our parameterization of texture and human pose are widely applicable and can easily be extended to generate animations based on learned humanoid control policies and hence can be utilized for video generation.

Furthermore, the textures generated by our method are compatible with existing graphics engines such as Blender, Unity and Unreal Engine and hence can be utilized by animators and content creators for a wide range of applications.

Committee:

Kris Kitani (Advisor)

Katerina Fragkiadaki

Ye Yuan