Abstract:  Generating and transforming content requires both creativity and skill. Creativity defines what is being created and why, while skill answers the question of how. While creativity is believed to be abundant, skill can often be a barrier to creativity. In our team, we aim to substantially reduce this barrier. Recent Generative AI methods have simplified the problem for 2D designers and illustrators. In this seminar, I will discuss our work towards realistic and intuitive 3D content creation. First, I will present a 3D generator that supports all ImageNet object categories at the same time. Second, we will look at a new 3D animation framework that is trained purely on monocular videos and capable of reconstructing shape, normals, color, and articulation parameters for a wide range of object categories. A single model can generate new objects, synthesize novel views and animate at the same time. Finally, we will discuss the concept of learnable game engines and see their applications in playing games and manipulating real-world videos of 3D games using text prompts, styles, and camera trajectories. Our model learns the geometry and physics of the scene and most excitingly supports game AI, allowing the user to specify a high-level goal for the game AI to execute.

Bio: Sergey Tulyakov is a Principal Research Scientist at Snap Inc. He leads the Creative Vision team and focuses on creating methods for transforming the world via computer vision and machine learning. His work includes 2D and 3D synthesis, photorealistic manipulation and animation, video synthesis, prediction and retargeting. Sergey pioneered the unsupervised image animation domain with MonkeyNet and First Order Motion Model that sparked a number of startups in the domain. His work on Interactive Video Stylization received the Best in Show Award at SIGGRAPH Real-Time Live! 2020. He has published 40+ top conference papers, journals and patents resulting in multiple innovative products, including Snapchat Pet Tracking, OurBaby, Real-time Neural Lenses, recent real-time try-on and many others. Before joining Snap Inc., Sergey was with Carnegie Mellon University, Microsoft, NVIDIA. He holds a PhD degree from the University of Trento, Italy.

Homepage:  http://www.stulyakov.com/

Sponsored in part by:   Meta Reality Labs Pittsburgh