Abstract:
How can we enable robots to perceive, adapt, and understand their surroundings like humans—in real-time and under uncertainty? Just as humans rely on vision to navigate complex environments, robots need robust and intelligent perception systems—“eyes” that can endure sensor degradation, adapt to changing conditions, and recover from failure. However, today’s visual systems are fragile—easily disrupted by occlusion, lighting variation, motion blur, or dynamic scenes. A single sensor dropout or unexpected environmental change can trigger cascading errors, severely compromising autonomy.

In this talk, I’ll share my journey in developing resilient and intelligent visual perception systems—building the “robust eyes” robots need to thrive in the real world.

The first part of the thesis introduces my prior work, beginning with Super Odometry, a unified sensor fusion pipeline that seamlessly integrates diverse modalities. We then explore methods to estimate uncertainty in both visual (MSO) and geometric (SuperLoc) degradation scenarios. To further enhance resilience, I present a hierarchical adaptation strategy that enables recovery from failure—featuring TartanIMU and Adaptive SLAM. At the heart of this work is a paradigm shift: elevating the IMU from a supporting sensor to the core estimator, with visual and LiDAR data adaptively reinforcing or correcting the inertial model.

The second part of the thesis outlines my proposed research on integrating foundation model priors into SLAM. First, I present SuperMap, which brings semantic priors into the SLAM pipeline to enable open-vocabulary, 4D object-level mapping—allowing robots to detect, track, and understand objects over time. Second, I introduce SparseVIO, a method that incorporates learned 3D reconstruction and motion priors to improve SLAM robustness in perceptually degraded and dynamic environments.

Together, these efforts form a unified framework for building resilient, intelligent, efficient SLAM systems—designed to work for any robot, anywhere.

Thesis Committee Members:
Sebastian Scherer (Chair)
Michael Kaess
Shubham Tulsiani
Jakob Engel (Meta AI)