Abstract:
As collaborative robots are increasingly deployed in personal environments, such as the home, it is critical they take actions to complete tasks consistent with personal preferences. Determining personal preferences for completing household chores, however, is challenging. Many household chores, such as setting a table or loading a dishwasher, are sequential and open-vocabulary, creating a landscape of almost endless a priori preferences. Taking assistive actions in such a domain means that a robot must first determine someone’s personal preference from within this expansive space. To do this, robots rely on people to communicate information about their preferences.Communication about preferences is often collected ex-situ: a person is presented with an abstract situation with several alternative solutions and gives feedback about which solution they think they would prefer if they were acting in-situ. This feedback about the preferred solution, combined with similar responses from multiple people in multiple situations, is then used to train a preference model. This data can be burdensome to collect, relies on ex-situ data collection which does not guarantee alignment with in-situ preferences, and fails to capture information about changing to preferences that may arise due to the execution of the collaboration.

In this thesis, we argue that robots can provide assistance personalized in-situ using observations of naturalistic human behaviors. In other words, robotic assistance can be viewed as a process of value alignment and can be achieved during task execution using observations of naturally occurring goal-directed behaviors. To support this argument, we make five main contributions.

First, we define assistive robotics as a value alignment problem and identify the main components in defining such a problem: the people involved, the space (or environment) in which the interaction takes place, and the relative timing of the robot and collaborative partners’ actions. Second, we introduce a dataset of naturalistic human robot collaboration behavior collected in a simple collaborative object rearrangement task. Third, we use this dataset to highlight the importance of continued personalization in assistive scenarios. Fourth, we present a method for extending these ideas to complex surface rearrangement tasks with naturalistic data using large internet-scale pretrained multi-modal foundation models. Finally, we present a method for continually finetuning these large foundation models using naturalistic, in-situ behaviors, demonstrating how we can provide seamless robotic assistance from varying sources of in-situ human behavior data.

Thesis Committee Members:
Henny Admoni, Chair
Kris Kitani, Co-Chair
Andrea Bajcsy
Dylan Losey, Virginia Polytechnic Institute and State University
Christopher Paxton, Hello Robot