Event Location: GHC 4405

Abstract: Commanding robots through unconstrained natural language directions is intuitive, flexible, and does not require specialized interfaces or training. Providing this capability would enable effortless coordination in human robot teams that operate in non-specialized environments. However, natural language direction following through unknown environments requires understanding the structure of language, mapping verbs and spatial relationships onto actions in the world, recognizing diverse landmarks located in the environment, as well as reasoning about the environment and landmarks that have not yet been detected.

We address the problem of robots following natural language directions through complex unknown environments. By exploiting the structure of spatial language, we can frame direction following as a problem of sequential decision making under uncertainty. We learn a policy which predicts a sequence of actions that follow the directions. The policy explores the environment (discovering landmarks), backtracks when necessary, and explicitly declares when it has reached its destination. We use imitation learning to train the policy, using demonstrations of people following directions. By training explicitly in unknown environments we can generalize to situations that have not been encountered previously.

To robustly follow complicated or ambiguous directions, we propose an information space representation of direction following through unknown environments. By estimating the distribution over possible information states (beliefs), we propose to estimate our confidence while following directions, and more importantly keep track of multiple hypotheses. We also propose to plan directly in this information space, in order to actively reduce the uncertainty of our belief distribution and maximize the likelihood of reaching the correct destination.

Generally, this work proposes to enable robots with imperfect perception to autonomously follow directions expressed in natural language through unknown environments.

Committee:Anthony Stentz, Chair


J. Andrew Bagnell


Manuela M. Veloso


Nicholas Roy, Massachusetts Institute of Technology