Towards Large-Scale and Long-Term Neural Map Representations

Abstract

We address the problem of large-scale and long-term neural map representations. Maps provide valuable information for modern robotic applications such as autonomous driving and AR/VR. In this thesis, we explored two important perspectives of map design: size and richness. First, we look into the map compression problem for image-to-LiDAR map, LiDAR-to-LiDAR map, and image-to-SfM map registration. For image-to-LiDAR map registration, we proposed a learning-based technique to precompute and compress a voxelized LiDAR map before performing image registration. For LiDAR-to-LiDAR map registration, we performed map compression benchmarks for existing deep learning based and traditional methods. For image-to-SfM map registration, we proposed selecting important keypoints from a SfM map through a heterogeneous graph neural network. The outcomes of all the three works lead to significant reduction of map size with offline preprocessing, and thus offloads the data burden of online image registration.
Second, inspired by the promising results of recent NeRF works, we developed a LiDAR-assisted NeRF system that encodes the rich appearance and geometry details of an outdoor environment into point-based neural representation and performs novel view synthesis. Unlike most of the previous NeRF works that focus on indoor or small scenes, our system is designed for more challenging canonical autonomous driving datasets such as Argoverse 2, which has scarcer training views and larger scene complexity. We use a point-based NeRF framework with a conditional GAN, and successfully outperformed state-of-the-art outdoor NeRF baselines. In addition, we explored several applications for outdoor NeRFs, including data augmentation, object detection, and seasonal view synthesis. Our experiments show the foreseeable potential of applying neural representation for more practical outdoor applications in the future.