Event Location: NSH 1507

Abstract: As robots become more reliable and user interfaces (UI) become more powerful, human-robot-agent teams are being applied to an increasing number of real world problems. Common areas of interest for these teams include scientific investigation, surveillance, disaster response, and search and rescue. These systems often employ a set of high level team plans to describe the supervisory roles of a human operator(s) and the actions of the robotic platforms. Agents in these teams provide a number of services to assist the operator with responsibilities such as task allocation, path planning, and visual object recognition. The UI provided to the operator usually implements human factors techniques that enhance the operator’s situational awareness (SA) during plan execution and adjust mixed initiative (MI) invocation of agent services to manage the operator’s workload. A system with the right set of team plans, SA and MI techniques, and invocation settings for agents often allows the operator to effectively utilize dozens of robots. However, these techniques and settings are typically static and do not adapt to changes in the plan’s context or the overall system configuration. Human factor strategies and agent service settings for one domain, device, or section of a plan may not be appropriate for others. In addition, different users and deployment locations may also require different settings, which are difficult to anticipate.

I propose to address these shortcomings in my thesis by augmenting team plans with situational awareness and mixed initiative (SAMI) markup, designed to be domain and device independent. The markup system allows experts to define settings at specific points in a plan concerning human factors techniques applied to UI elements, mixed initiative autonomy used during decision making, and performance constraints used by agents to choose service algorithms. Furthermore, the system will be capable of learning usage models of the team plans for particular users and locations and modifying the plan’s markup to improve performance for that setting.

In recent work I have developed a Petri Net based team plan language with mechanisms for task allocation and error recovery contingencies. SAMI markup is applied to events within a plan and is interpreted at runtime to select the most appropriate UI components and service algorithms for the event from the core and domain specific libraries. I have implemented a laptop GUI capable of dynamically constructing interaction panels for the operator using UI components and widgets from the libraries which satisfy the event and markup requirements. The team plan and markup language has been applied to and informally validated in two separate domains to control unmanned aerial and surface vehicles. A developed IDE assists in the development of these SAMI plans by using Petri Net theory to automate certain graph based actions and detect illegal or unexpected grammar.

Moving forward, I propose to continue developing the SAMI team plan and markup language to address shortcomings identified during extensive field testing of a team of robotic watercraft. Future work will also address the challenge of developing plan usage models for specific users and locations. Users may have different skill levels, personalities, dedicated supervision time, and team sizes which are best handled by different markup. Deployment locations may have topography or other characteristics which are best addressed by different service algorithms, error recovery methods, and mixed initiative settings. The research questions that will be addressed in this portion of the thesis include (1) how to extract coherent features from raw input and event stream data, (2) how to learn models quickly based on a limited amount of data, and (3) how to generate test data for a wide variety of locations and users from a small set of collected data. Finally we must evaluate how well the markup system is able to capture desired behavior and how well the usage modeling is able to adapt team plans. These evaluations will involve a combination of metrics including required operator attention, robot performance, and task performance. The desired result of this work is increased system performance due to manual and learned plan markup capturing context specific settings and decreased development time for HRA systems due to the language’s conciseness and flexibility and the reuse of UI components and service algorithms.

Committee:Paul Scerri, Chair
Illah Nourbakhsh
Reid Simmons
Julie Adams, Vanderbilt University