Gaussian Representations for Differentiable Rendering and Optimization

Abstract

In this thesis, we explore some fast, robust and efficient methods that are generally useful across many domains. Specifically, we explore the use of Gaussian Representations in multiple application areas of computer vision and robotics.

In the first part, we provide an alternative approach to the classic hidden surface problem. In particular, we design a ray-based differentiable renderer for 3D Gaussians that can be used to solve multiple classic computer vision problems in a unified manner. For example, we can reconstruct 3D shapes from color, silhouette or optical flow, based solely on gradient-based optimization; these reconstructions are robust to input errors and reasonably fast (taking a few minutes on a laptop CPU). Similarly, we can solve for precise camera pose estimates for known objects, comparable to the quality given by classic methods. Our contributions include an alternative formulation of the hidden surface problem that sacrifices fidelity for utility, thereby obtaining fast runtimes and high-quality gradient information. We extend this renderer with differentiable optical flow and show how to export colored meshes from the reconstructions. We show examples on naturally collected videos of everyday objects.We will also cover our work on obtaining 3D Gaussian representations directly from meshes, without the need for sampling point sets.

In another line of our research, we show how Gaussian representations provide a powerful underlying representation for gradient-free optimization of classic algorithms in robotics such as stereoscopic depth matching, motion planning, visual odometry, and social navigation. We develop techniques for performing optimization based on user preferences and based on dataset variation. We show how Gaussian representations can be tuned directly from user preferences, without the need for ground-truth collection or fine-tuned metric design. Additionally, we show these optimizers can discover multiple algorithm configurations for potentially different environments, based solely on the algorithm responding to sampled configurations.

Lastly, we will touch on a novel regression-based, citation-free alternative to citation metrics for analyzing academic contributions.