Abstract:
The value alignment problem considers how robots can learn to behave in accordance with human values. Today, robot learning paradigms enable humans to provide data (e.g., preference labels or demonstrations), which the robot uses to update its behavior (e.g., reward model or policy) to be closer to the human’s values. However, the current paradigm requires the user to constantly supervise, provide new feedback, and—more fundamentally—perfectly understand where the robot is misaligned. Even if the robot eventually learns a perfect model of how the user wanted it to behave, the overall human-robot interaction during alignment could have been demanding, confusing, or arduous for the person.

This dissertation proposes that alignment isn’t just about a robot’s understanding of the task; it must also account for the overall interaction during the alignment process. In other words, alignment shouldn’t be treated as just a destination; it’s a journey. To achieve this goal, we break down the value alignment process into two levels: task alignment (wherein the robot understands and behaves in accordance with the human’s goals and intents) and interaction alignment (wherein the robot communicates with and seeks feedback from the user in accordance with the user’s interaction preferences). In our completed work, we enable task alignment in multi-agent collaborative games via online strategy adaptation and propose a way for shared robot controllers to measure their misalignment via conformal prediction. We further demonstrate a first step towards interaction alignment via proactive robot strategy explanations. In our proposed work, we will (1) formalize how different types of uncertainty (ie. action uncertainty, ambiguous human instruction, skill failure) inform different interactive, in-the-moment queries to the user, and will (2) mathematically model interaction-alignment over both physical and communicative robot behaviors. Our ultimate goal is to enable robots that are proactive participants in the value alignment process: robots that seek feedback strategically when they are uncertain during task execution and are cognizant of the capacity for human feedback constrained by the human’s state and the nature of the task.

Thesis Committee Members:
Henny Admoni, Co-chair
Reid Simmons, Co-chair
Andrea Bajcsy
Anirudha Majumdar, Princeton University

More Information