Speech and gaze conflicts in collaborative human-robot interactions

Abstract

Gaze and speech are both important modes of communication for human-robot interactions. However, few studies to date have explored the effects of conflict in a robot’s multi-modal communication. In this paper, we investigate how such speech-gaze conflicts affect performance on a cooperative referential task. Participants play a selection game with a robot, in which the robot instructs them to select one object from among a group of available objects. We vary whether the robot’s gaze is congruent with its speech, incongruent with its speech, or absent, and we measure participants’ response times to the robot’s instructions. Results indicate that congruent speech facilitates performance but that incongruent speech does not hinder performance. We repeat the study with a human actor instead of a robot to investigate whether human gaze has the same effect, and find the same results: in this type of activity, congruent gaze helps performance while incongruent gaze does not hurt it. We conclude that robot gaze may be a worthwhile investment in such situations, even when gaze behaviors may be unreliable.