Abstract:
Enabling robots to perform complex dynamic tasks such as picking up an object in one sweeping motion or pushing off a wall to quickly turn a corner is a challenging problem. The dynamic interactions implicit in these tasks are critical for successful task execution. Furthermore, given the interactive nature of such tasks, safety, in the context of ensuring safe contacts, is a necessary consideration. In this thesis, we focus on developing machine learning paradigms that enable robots to safely perform dynamic interactive tasks by reasoning about interactions while ensuring generalizability of the learned behaviors.

In this talk, I will discuss my work on exploring the use of structural priors (SLDS and GNNs) for explicitly modeling the switching dynamics of such multi-body systems. Furthermore, I will discuss the use of algorithmic priors (LQR) for learning predictive and reactive differentiable policies for real time control over multiple dynamic modes. Our experiments demonstrate that incorporating such priors results in learned models and skills that can generalize to more complex environments and tasks. My more recent work focuses on learning to perform dynamic and precise tasks without relying on ground truth state information but by incorporating multimodal sensing. We observe that such tasks require sensing at different spatial and temporal resolutions for successful task completion. Thus, we propose a multi-resolution (spatial and temporal) sensor fusion approach that utilizes networks of varying capacities to effectively combine global and local spatial information at different frequencies. Our learned policy generalizes to semantic variations in the scene while also enabling real time control of precise and reactive tasks. Finally, I will talk about my proposed work on developing safety-aware interactive agents by formulating safety in the context of ensuring safe contacts with the environment. The aim of this proposed work is to develop safety-aware instruction-following agents by combining the generalizability of large pre-trained vision-language models with the theoretical guarantees of algorithmic priors.

Thesis Committee Members:
Oliver Kroemer, Chair
Aaron Johnson
Zico Kolter
Michael Posa, University of Pennsylvania

More Information