Evaluation of a Remote Data Collection Method to Study Human-Automation Interaction and Workload

Meghan Chandarana; Eric T Chancey; Lisa R Le Vie; Michael S Politowicz

NASA NTRS · 20220001930

Evaluation of a Remote Data Collection Method to Study Human-Automation Interaction and Workload

Meghan Chandarana·Eric T Chancey·Lisa R Le Vie·Michael S Politowicz

Abstract

Technological advances have increased the automation of Uncrewed Aerial Vehicles, allowing human operators to manage multiple vehicles at a high-level without the need to understand low-level system behaviors. Previous laboratory studies have explored the relationship between reliability, trust, use of automation and the effects of number of vehicles under supervision on subjective workload. Due to the limitations resulting from the COVID-19 pandemic, in-person laboratory studies are not always possible. Therefore, this work aimed to investigate if remote data collection alternatives, such as Amazon’s Mechanical Turk, can provide comparative results as those obtained in laboratory settings. A study was conducted in the context of small drone operations. As expected, higher reliability led to higher trust ratings and the inclusion of more vehicles led to higher workload. In contrast, reliability unexpectedly had no effect on intention to use the automation. Though these results were encouraging, several limitations were identified.

Explore related subjects

trust reliance human-automation interaction remote data collection

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Meghan Chandarana, Eric T Chancey, Lisa R Le Vie, Michael S Politowicz. Evaluation of a Remote Data Collection Method to Study Human-Automation Interaction and Workload. https://ntrs.nasa.gov/citations/20220001930

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

NASA NTRS2016

Effects of Transparency on Pilot Trust and Agreement in the Autonomous Constrained Flight Planner

We performed a human-in-the-loop study to explore the role of transparency in engendering trust and reliance within highly automated systems. Specifically, we examined how transparency impacts trust in and reliance upon the Autonomous Constrained Flight Planner (ACFP), a critical automated system being developed as part of NASA's Reduced Crew Operations (RCO) Concept. The ACFP is designed to provide an enhanced ground operator, termed a super dispatcher, with recommended diversions for aircraft when their primary destinations are unavailable. In the current study, 12 commercial transport rated pilots who played the role of super dispatchers were given six time-pressured all land scenarios where they needed to use the ACFP to determine diversions for multiple aircraft. Two factors were manipulated. The primary factor was level of transparency. In low transparency scenarios the pilots were given a recommended airport and runway, plus basic information about the weather conditions, the aircraft types, and the airport and runway characteristics at that and other airports. In moderate transparency scenarios the pilots were also given a risk evaluation for the recommended airport, and for the other airports if they requested it. In the high transparency scenario additional information including the reasoning for the risk evaluations was made available to the pilots. The secondary factor was level of risk, either high or low. For high-risk aircraft, all potential diversions were rated as highly risky, with the ACFP giving the best option for a bad situation. For low-risk aircraft the ACFP found only low-risk options for the pilot. Both subjective and objective measures were collected, including rated trust, whether the pilots checked the validity of the automation recommendation, and whether the pilots eventually flew to the recommended diversion airport. Key results show that: 1) Pilots trust increased with higher levels of transparency, 2) Pilots were more likely to verify ACFPs recommendations with low levels of transparency and when risk was high, 3) Pilots were more likely to explore other options from the ACFP in low transparency conditions and when risk was high, and 4) Pilots decision to accept or reject ACFPs recommendations increased as a function of the transparency in the explanation. The finding that higher levels of transparency was coupled with higher levels of trust, a lower need to verify other options, and higher levels of agreement with ACFP recommendations, confirms the importance of transparency in aiding reliance on automated recommendations. Additional analyses of qualitative data gathered from subjects through surveys and during debriefing interviews also provided the basis for new design recommendations for the ACFP.

trust↗

NASA NTRS2014

Influence of Cultural, Organizational, and Automation Capability on Human Automation Trust: A Case Study of Auto-GCAS Experimental Test Pilots

This paper discusses a case study that examined the influence of cultural, organizational and automation capability upon human trust in, and reliance on, automation. In particular, this paper focuses on the design and application of an extended case study methodology, and on the foundational lessons revealed by it. Experimental test pilots involved in the research and development of the US Air Force's newly developed Automatic Ground Collision Avoidance System served as the context for this examination. An eclectic, multi-pronged approach was designed to conduct this case study, and proved effective in addressing the challenges associated with the case's politically sensitive and military environment. Key results indicate that the system design was in alignment with pilot culture and organizational mission, indicating the potential for appropriate trust development in operational pilots. These include the low-vulnerability/ high risk nature of the pilot profession, automation transparency and suspicion, system reputation, and the setup of and communications among organizations involved in the system development.

trust↗

NASA NTRSDate not supplied

Evaluation of a Remote Data Collection Method to Study Human-Automation Interaction and Workload

Introduction - Paradigm shift from one operator supervising a single vehicle, to an operator supervising multiple highly automated vehicles (One-to-Many) - One-to-Many application - Search and Rescue - Foraging - Military ops - Etc. - Calibrated trust in automation enables human operators to effectively manage highly automated vehicles - Past studies show that trust mediates relationship between reliability and dependence (Chancey et al., 2017; Chancey et al., 2015) - Future studies needed to further understand relationship

trust↗