Abstract:
Large pre-trained models and internet data sources are key to general and efficient robot task learning. However, learning contact-rich behaviors, semantic task constraints, and robust task planning from internet data sources remains an open challenge. This proposal seeks to make progress towards a general robot task learning system leveraging pre-trained models and internet data. We limit our study to tasks in the cooking and food preparation domain. This proposal discusses our work in two key areas: skill learning and task planning. To improve skill learning, we present an approach for selecting between basic robot behaviors to accomplish cooking skills. We then propose further work to learn these basic robot behaviors and reduce the need for real-world execution when performing behavior selection. To improve task planning we conduct a large-scale study on Large Language Models as task planners. We identify key deficiencies and propose a new planning framework to address them. Finally, we propose a method for learning task ordering constraints from human video based on our prior constraint-learning work. Ultimately, we seek to produce a robot task-learning system capable of quickly acquiring new tasks in a home environment.

Thesis Committee Members:
Christopher Atkeson, Chair
David Held
Oliver Kroemer
Ruta Desai. Meta, FAIR

More Information