VP4D: View Planning for 3D and 4D Scene Understanding

Abstract

View planning plays a critical role in gathering views that optimize scene reconstruction, which is essential in virtual production and computer animation. A 3D map of the film set and motion capture of actors lead to an immersive experience. Current methods use uncertainty estimation in the neural rendering of view candidates to avoid updating a 3D map, but these are limited to smaller scenes. Moreover, existing techniques for 4D motion capture struggle with obstacles, occlusions, and multiple actors. VP4D addresses these challenges by considering practical circumstances for gathering views of larger 3D scenes and accounting for occlusions and obstacles while filming multiple actors in 4D scenes. It presents: (i) Data curation methods to form image clusters that enable uncertainty estimation-based view planning for large-scale outdoor unknown scenes. This includes a clustering method for outdoor scenes using a similarity matrix computed with Structure from Motion (SfM) for stable training performance; (ii) Multi-view planning methods that address obstacles and occlusions using drone-mounted cameras for multiple moving actors in a known scene. This involves sequential and coordinated multi-view planning methods that capture the moving actors in the 4D scene, ensuring view diversity and pixel coverage. Together, these techniques provide novel view planning methods that enhance the quality of views gathered for both 3D and 4D outdoor scene reconstruction applications.