Abstract:
Two decades into the third age of AI, the rise of deep learning has yielded two seemingly disparate realities. In one, massive accomplishments have been achieved in deep reinforcement learning, protein folding, and large language models. Yet, in the other, the promises of deep learning to empower robots that operate robustly in real-world environments have yet remained unfulfilled. Vast diversity of objects, distribution shifts, long-tailed phenomena: outside of laboratories, real-world environments challenge modern statistical learning assumptions of the data.

Although such environments have generally been referred to as “unstructured,” this terminology belies their nature. Real-world environments are not “unstructured,” but arise because of structure: the underlying causal processes that generate the observed data. In this view, robots should not only reason and learn with respect to data, but also the data generating processes. Such processes can be formalized by the language of causality. Therefore, to learn and leverage the structure of these “open-world” environments, new causal-based robot learning algorithms are needed.

Towards this end, this thesis explores new avenues in causal learning and reasoning for a diversity of robot learning problems, from perception to control. First, we explore how causal features can be learned for perception models using sim-to-real transfer from synthetic data (DREAM, FormNet). We continue this theme in the introduction of structural sim-to-real transfer, where simulation can serve as a causal reasoning engine for the robot to select the relevant features for a control policy (CREST) or skill (SCALE). By construction, these policies are robust to irrelevant distribution shifts that would otherwise stymie correlation-based deep learning. Next, the rich interplay between control, dynamical systems, and causality is explored through the Learning By Doing (LBD) competition and LMeshNet, a methodology for constructing hybrid causal world models that integrate both latent and semantic information. Lastly, in causal curriculum learning, we first explore the commonalities and differences between human and agent curriculum learning (ACL). Then, we employ these lessons learned for CURATE: how agents can perform interventions to manipulate the sequencing of training data to efficiently learn a control policy to solve a desired, difficult task.

The advantages of adopting the principles of causal inference have been witnessed to date in fields such as biomedical sciences, economics, and genomics. In the machine learning community, it has recently been argued that such principles should be integrated to harness deep learning, towards causal learning of representations. Analogously, this thesis forwards that the robot learning community stands to gain by leveraging the principles of causality. In so doing, this new paradigm holds promise for robots to learn and leverage structure within the open world through causal robot learning for manipulation.

Thesis Committee Members:
Oliver Kroemer, Chair
Shubham Tulsiani
Kun Zhang
Jonas Martin Peters, ETH Zürich

More Information