PROJECT OVERVIEW

What did the autonomous robot vehicle say to the human operator?

Organization

U.S. Army Ground Vehicle Systems Center (GVSC), CMU HCII

My Role

UX & Interaction Designer

Helping Robot Vehicles & Human Operators Solve Anomalies Together

Interaction Design

multimodal ui

The U.S. Army Ground Vehicle Systems Center (GVSC) is looking to equip human teams with autonomous robot vehicles that can enhance field operations and support teams in labor intensive tasks. Working in tandem with the engineering team, I led the design team to create the GVSC Robot Anomaly Interface (RAI) system to facilitate efficient communication between operators and their robot vehicles.

Timeline

January - May 2021

I was responsible for...

interaction design

UX/UI

prototyping

info architecture

User testing

the team

Advisors

Nikolas Martelaro
Carol Smith

Design Team

Malika Khurana
Serena Wang

Engineering Team

David Perry
Humphrey Yang

The Challenge

Robots, Operators, & Anomalies

The GVSC's autonomous robot vehicles are used in road reconnaissance operations to gather data on terrain, obstacles, and other environmental info on predetermined routes.

During operations, robots often encounter anomalies, or abnormal events that can be triggered by a multitude of different environmental and internal robot system issues during operations. In these situations, human operators are often faced with the challenge simultaneously monitoring and controlling their robots while multitasking in chaotic environments.

The Solution

The RAI System for Anomaly Communication

Working closely with the engineering team, I led the design team to create a system of interfaces that leverages multiple modalities to reduce the time and effort required for GVSC operators to evaluate, diagnose, and troubleshoot anomalies.

Haptic smartwatch

Alerts & Quick Evaluation

The haptic smartwatch serves two main purposes: the haptic wristband is able to catch the operator’s attention to notify them of anomalies, even in noisy and chaotic environments.

11’ Tablet

Diagnosing & Troubleshooting

The tablet interface supports operators in in-depth diagnosing and troubleshooting robot anomalies. The tablet interface aims to present actionable data in a way that minimizes the operator’s cognitive load so they can maintain situational awareness while addressing anomalies as efficiently as possible.

Jump to design process -->

RESEARCH PROCESS

Understanding Operators, Autonomous Robots & Anomalies

User Needs & Design Goals

What do Operators Need from Alert Interfaces?

SEMI-STRUCTURED INTERVIEWS

Secondary research

Using insights from interviews with GVSC personnel, I created a framework that organized the operator’s process in addressing anomalies into three-stages: Alert, Interpret, and React. To ensure that the interfaces would be able to meet operator needs at each stage, I established the core design goals the interfaces would have to follow based on this framework:

stages of the operator's process

1. Alert

The operator is notified when the robot vehicle encounters an anomaly

2. interpret

The operator evaluates the anomaly in order to determine if they need to take further action

3. react

The operator chooses the best course of action to address the anomaly.

PRESERVE

Cognitive Load

Minimize cognitive load so operators can maintain situational awareness even in chaotic environments

Enable

Quick Evaluation

Present information clearly so operators can quickly understand the robot’s situation

Guide

Decision-making

Support operators in determining what the best course of action is for a given anomaly

design goals

Task Analysis

Capturing All Possible Anomalies

Task analysis

User scenarios

Using the Alert, Interpret, and React framework as a basis, I conducted a task analysis to map out different possible anomaly scenarios.

This helped our team gain a clearer picture of the actions operators would take at each stage in different scenarios, and the data they would need.

CUSTOMER JOURNEY MAPPING

Telling The Operator’s Story

Personas

Customer Journey maps

To further synthesize our research and build a more holistic understanding of the operator’s perspective, I identified key user scenarios and created personas and customer journey maps to more holistically illustrate the operator’s process in addressing different types of anomaly situations.

Design Process

Translating Operator Needs into Actionable Interfaces

IDEATION

Exploring Interfaces & Devices

Sketching

storyboarding

Having established our understanding of the operator’s needs, situations, and environment, we explored and compared different interface systems and combinations to identify what would best fit the operator’s needs.

Here, we took into consideration screen size, device capability, portability, and the relationships between different interfaces.

The Winners

Over the course of three iterations of sketches and user testing, we determined that a haptic smartwatch and tablet interface system would be the ideal fit for what operators needed:

haptic smartwatch

Of all the interfaces, the haptic smartwatch was best-suited to alerting operators in anomaly situations, allowing them to quickly evaluate anomalies.

In testing, operators liked that it was easily accessible and portable, and how it could effectively capture their attention via haptics even in loud, chaotic environments.

11" TABLET

User testing also revealed that a larger visual interface would be necessary in displaying the rich information operators need to diagnose and troubleshoot anomalies.

In this case, we determined that a tablet would be the best fit in maximizing portability, with an optimal screen size of 11 inches.

USER flow Overview

1. receive ALERT

The operator receives an anomaly alert, and quickly evaluates to determine if further action is needed.

Smartwatch

2. choose RESPONSE

If action is not required, operators can dismiss the alert or flag it for later review for technicians.

Smartwatch

3. Diagnose

If action is required, the operator opens tablet interface to diagnose anomaly and determine if they need to troubleshoot.

Tablet

4. Troubleshoot

The operator follows the guidelines in tablet interface to troubleshoot the anomaly.

Tablet

Layout & Info Architecture

Iterating on Data Displays, Functionality, & Visual Layout

wireframing

prototyping

usability testing

Across 6 iterations, I worked with my team to design and test the interfaces with the goal of meeting operator needs at each stage of the user flow. My main focus in this process was in optimizing the layout with limited screen real-estate and the information architecture of complex data components.

Maximizing Screen Real-estate

Given its limited screen real-estate, I designed the smartwatch to display only salient and actionable info to minimize cognitive load.

Hierarchy, Data, & Visualizations

To find the ideal structure of data components within the tablet interface, I explored and tested different layouts with users to determine the content, hierarchy, and screen real-estate of each component.

interaction design

Optimizing for Usability via Microinteractions

I designed the interactions in the interfaces with two main goals in mind: to ensure operators can quickly access the information they need in any given anomaly situation, and to minimize the effort required for them to control the interfaces.

Haptic Smartwatch

Alert notification

RESPONSE Options

system OVERVIEW

11" Tablet

Future impact

Measuring Success for GVSC Operators & Robots

metrics for success

Minimizing Time, Effort, and Risk in Anomaly Situations

Ultimately, the goal of this project is to reduce risk and complications for both operators and their robots, and, in turn, increase the rate of successful operations for the GVSC. To measure the interfaces’ effectiveness in achieving this goal, I worked with my team to establish the following set of success metrics:

If the interfaces are successful...

1. Average time

The average time operators spend in addressing an anomaly should decrease

2. rate of succcess

There should be an increased rate of success in diagnosing and troubleshooting anomalies

3. complications

There should be a reduction in anomaly-related complications, such as damage to robots, operator injuries