Abstract:
Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception. Typically these methods rely on simple models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional observations. They must in turn infer states that are used downstream by planning and control modules. These modules expect states that are both physically plausible and efficient to query. In this thesis, we propose a set of approaches to incorporate richer observation models, physics models, and efficient maps within factor graph-based inference for robot perception. We ground our approaches in an application where a robot hand must infer object states during manipulation from tactile image measurements.

We address three principal challenges: (1) Maps generated from perception are also used by planning and must include free space information efficient to both update and query. (2) Variables in the graph typically model real-world physical quantities for which certain physics-based assumptions cannot be violated. We model these assumptions as hard constraints in the optimization and develop a novel incremental constrained algorithm to solve for this objective. (3) Graph optimizers typically rely on analytic observation models that are challenging to engineer for high-dimensional measurements such as tactile images. We circumvent this challenge by learning such models directly from tactile data and integrating them as factors in the graph optimizer.

For the remainder of this thesis, we propose the following extensions to completed work: To address Challenge 3, we propose to learn observation models that improve end-to-end tracking performance. Our key insight is to couple learning and optimization so that the learner extracts the most salient information from measurements. To address Challenge 2, we propose to extend our incremental constrained solver to handle re-linearizations on-the-fly to tackle highly nonlinear problems. Finally, we propose to apply these richer models for tracking object states during manipulation tasks such as planar pushing and in-hand object tracking under significant visual occlusions.

More Information

Thesis Committee Members:
Michael Kaess, Chair
David Wettergreen
Oliver Kroemer
Stuart Anderson, Facebook AI Research
John Leonard, Massachusetts Institute of Technology