Abstract:
View planning plays a critical role by gathering views that optimize scene reconstruction. Such reconstruction has played an important part in virtual production and computer animation, where a 3D map of the film set and motion capture of actors lead to an immersive experience. Current methods use uncertainty estimation in neural rendering of view candidates to bypass updating a 3D map; however, these methods are limited to smaller scenes. Additionally, for motion capture of moving actors, existing methods lack the ability to reason about obstacles and occlusions in the scene and do not account for multiple actors. VP4D presents (i) data curation methods to form image clusters that enable such uncertainty estimation-based view planning for large-scale outdoor unknown scenes, and (ii) multi-view planning methods that address obstacles and occlusions using drone-mounted cameras for multiple moving actors in a known scene. We present a clustering method for outdoor scenes by a similarity matrix computed using Structure from Motion (SfM) that leads to a stable training performance. We then present a sequential and coordinated multi-view planning method that captures the moving actors in the 4D scene ensuring view diversity and pixel coverage. Together, these provide novel view planning techniques crucial for both 3D and 4D outdoor scene reconstruction applications.

Committee:
Sebastian Scherer (advisor)
Katia Sycara (advisor)
Jiaoyang Li
Cherie Ho