Jonathon Luiten
Postdoctoral Fellow
RWTH Aachen and Carnegie Mellon University
Monday, November 6
3:30 pm to 4:30 pm
Newell-Simon Hall 3305
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
Abstract: We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians’ motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.
Bio: Jonathon Luiten is about to graduate with his PhD from the RWTH Aachen in Germany working with Prof. Bastian Leibe. He has spent the last 2 years as a visitor at Carnegie Mellon University with Prof. Deva Ramanan and before that at the University of Oxford with Prof. Philip Torr. Jonathon is known for his work on object tracking, including state-of-the-art approaches for single-object tracking, multi-object tracking and video object segmentation. He has redefined the way tracking is evaluated and benchmarked a number of times with the use of segmentation masks, better tracking metrics, large diverse datasets and moving tracking to the open world. He is the organiser of a number of CVPR/ICCV/ECCV workshops and has won a number of computer vision challenges. Jonathon will be presenting his most recent work in which he combines tracking and novel-view synthesis techniques in a unified analysis-by-synthesis framework.
