Engineering topics

Shafto, Michael

Publications and source records attributed to Shafto, Michael.

At least 19 records

NASA NTRS2013

Aviation Safety: Modeling and Analyzing Complex Interactions between Humans and Automated Systems

The on-going transformation from the current US Air Traffic System (ATS) to the Next Generation Air Traffic System (NextGen) will force the introduction of new automated systems and most likely will cause automation to migrate from ground to air. This will yield new function allocations between humans and automation and therefore change the roles and responsibilities in the ATS. Yet, safety in NextGen is required to be at least as good as in the current system. We therefore need techniques to evaluate the safety of the interactions between humans and automation. We think that current human factor studies and simulation-based techniques will fall short in front of the ATS complexity, and that we need to add more automated techniques to simulations, such as model checking, which offers exhaustive coverage of the non-deterministic behaviors in nominal and off-nominal scenarios. In this work, we present a verification approach based both on simulations and on model checking for evaluating the roles and responsibilities of humans and automation. Models are created using Brahms (a multi-agent framework) and we show that the traditional Brahms simulations can be integrated with automated exploration techniques based on model checking, thus offering a complete exploration of the behavioral space of the scenario. Our formal analysis supports the notion of beliefs and probabilities to reason about human behavior. We demonstrate the technique with the Ueberligen accident since it exemplifies authority problems when receiving conflicting advices from human and automated systems.

Muti-agent System↗

NASA NTRS2013

Work Practice Simulation of Complex Human-Automation Systems in Safety Critical Situations: The Brahms Generalized berlingen Model

The transition from the current air traffic system to the next generation air traffic system will require the introduction of new automated systems, including transferring some functions from air traffic controllers to on-board automation. This report describes a new design verification and validation (V&V) methodology for assessing aviation safety. The approach involves a detailed computer simulation of work practices that includes people interacting with flight-critical systems. The research is part of an effort to develop new modeling and verification methodologies that can assess the safety of flight-critical systems, system configurations, and operational concepts. The 2002 Ueberlingen mid-air collision was chosen for analysis and modeling because one of the main causes of the accident was one crew's response to a conflict between the instructions of the air traffic controller and the instructions of TCAS, an automated Traffic Alert and Collision Avoidance System on-board warning system. It thus furnishes an example of the problem of authority versus autonomy. It provides a starting point for exploring authority/autonomy conflict in the larger system of organization, tools, and practices in which the participants' moment-by-moment actions take place. We have developed a general air traffic system model (not a specific simulation of Überlingen events), called the Brahms Generalized Ueberlingen Model (Brahms-GUeM). Brahms is a multi-agent simulation system that models people, tools, facilities/vehicles, and geography to simulate the current air transportation system as a collection of distributed, interactive subsystems (e.g., airports, air-traffic control towers and personnel, aircraft, automated flight systems and air-traffic tools, instruments, crew). Brahms-GUeM can be configured in different ways, called scenarios, such that anomalous events that contributed to the Überlingen accident can be modeled as functioning according to requirements or in an anomalous condition, as occurred during the accident. Brahms-GUeM thus implicitly defines a class of scenarios, which include as an instance what occurred at Überlingen. Brahms-GUeM is a modeling framework enabling "what if" analysis of alternative work system configurations and thus facilitating design of alternative operations concepts. It enables subsequent adaption (reusing simulation components) for modeling and simulating NextGen scenarios. This project demonstrates that BRAHMS provides the capacity to model the complexity of air transportation systems, going beyond idealized and simple flights to include for example the interaction of pilots and ATCOs. The research shows clearly that verification and validation must include the entire work system, on the one hand to check that mechanisms exist to handle failures of communication and alerting subsystems and/or failures of people to notice, comprehend, or communicate problematic (unsafe) situations; but also to understand how people must use their own judgment in relating fallible systems like TCAS to other sources of information and thus to evaluate how the unreliability of automation affects system safety. The simulation shows in particular that distributed agents (people and automated systems) acting without knowledge of each others' actions can create a complex, dynamic system whose interactive behavior is unexpected and is changing too quickly to comprehend and control.

complex systems↗

NASA NTRS2009

On Organization of Information: Approach and Early Work

In this report we describe an approach for organizing information for presentation and display. "e approach stems from the observation that there is a stepwise progression in the way signals (from the environment and the system under consideration) are extracted and transformed into data, and then analyzed and abstracted to form representations (e.g., indications and icons) on the user interface. In physical environments such as aerospace and process control, many system components and their corresponding data and information are interrelated (e.g., an increase in a chamber s temperature results in an increase in its pressure). "ese interrelationships, when presented clearly, allow users to understand linkages among system components and how they may affect one another. Organization of these interrelationships by means of an orderly structure provides for the so-called "big picture" that pilots, astronauts, and operators strive for.

Degani, Asaf↗

NASA NTRS2006

Statistical Detection of Atypical Aircraft Flights

A computational method and software to implement the method have been developed to sift through vast quantities of digital flight data to alert human analysts to aircraft flights that are statistically atypical in ways that signify that safety may be adversely affected. On a typical day, there are tens of thousands of flights in the United States and several times that number throughout the world. Depending on the specific aircraft design, the volume of data collected by sensors and flight recorders can range from a few dozen to several thousand parameters per second during a flight. Whereas these data have long been utilized in investigating crashes, the present method is oriented toward helping to prevent crashes by enabling routine monitoring of flight operations to identify portions of flights that may be of interest with respect to safety issues.

Statler, Irving↗

NASA NTRS2004

Comparing Methods for UAV-Based Autonomous Surveillance

We describe an approach to evaluating algorithmic and human performance in directing UAV-based surveillance. Its key elements are a decision-theoretic framework for measuring the utility of a surveillance schedule and an evaluation testbed consisting of 243 scenarios covering a well-defined space of possible missions. We apply this approach to two example UAV-based surveillance methods, a TSP-based algorithm and a human-directed approach, then compare them to identify general strengths, and weaknesses of each method.

Freed, Michael↗

NASA NTRS2002

Motion Planning in a Society of Intelligent Mobile Agents

The majority of the work on this grant involved formal modeling of human-computer integration. We conceptualize computer resources as a multiagent system so that these resources and human collaborators may be modeled uniformly. In previous work we had used modal for this uniform modeling, and we had developed a process-algebraic agent abstraction. In this work, we applied this abstraction (using CSP) in uniformly modeling agents and users, which allowed us to use tools for investigating CSP models. This work revealed the power of, process-algebraic handshakes in modeling face-to-face conversation. We also investigated specifications of human-computer systems in the style of algebraic specification. This involved specifying the common knowledge required for coordination and process-algebraic patterns of communication actions intended to establish the common knowledge. We investigated the conditions for agents endowed with perception to gain common knowledge and implemented a prototype neural-network system that allows agents to detect when such conditions hold. The literature on multiagent systems conceptualizes communication actions as speech acts. We implemented a prototype system that infers the deontic effects (obligations, permissions, prohibitions) of speech acts and detects violations of these effects. A prototype distributed system was developed that allows users to collaborate in moving proxy agents; it was designed to exploit handshakes and common knowledge Finally. in work carried over from a previous NASA ARC grant, about fifteen undergraduates developed and presented projects on multiagent motion planning.

Esterline, Albert C.↗

NASA NTRS2002

Dual-Task Interference When A Response is Not Required

When subjects are required to respond to two stimuli presented in rapid succession, responses to the second stimulus are delayed. Such dual-task interference has been attributed to a fundamental processing bottleneck preventing simultaneous processing on both tasks. Two experiments show dual-task interference even when the first task does not require a response. The observed interference is caused by a bottleneck in central cognitive processing, rather than in response initiation or execution.

VanSelst, Mark↗

NASA NTRS2001

On Abstractions and Simplifications in the Design of Human-Automation Interfaces

This report addresses the design of human-automation interaction from a formal perspective that focuses on the information content of the interface, rather than the design of the graphical user interface. It also addresses the, issue of the information provided to the user (e.g., user-manuals, training material, and all other resources). In this report, we propose a formal procedure for generating interfaces and user-manuals. The procedure is guided by two criteria: First, the interface must be correct, i.e., that with the given interface the user will be able to perform the specified tasks correctly. Second, the interface should be as succinct as possible. The report discusses the underlying concepts and the formal methods for this approach. Several examples are used to illustrate the procedure. The algorithm for constructing interfaces can be automated, and a preliminary software system for its implementation has been developed.

Heymann, Michael↗

NASA NTRS2001

Multiagent Modeling and Simulation in Human-Robot Mission Operations Work System Design

This paper describes a collaborative multiagent modeling and simulation approach for designing work systems. The Brahms environment is used to model mission operations for a semi-autonomous robot mission to the Moon at the work practice level. It shows the impact of human-decision making on the activities and energy consumption of a robot. A collaborative work systems design methodology is described that allows informal models, created with users and stakeholders, to be used as input to the development of formal computational models.

Sierhuis, Maarten↗

NASA NTRS1998

Formal Analysis of Human Automation Interaction: The Problem of Display Correctness

This presentation attempted to develop a quantitative method for evaluating the effectiveness of autoflight displays. Researchers sought to develop a methodology for evaluation of display correctness and develop a methodology for display synthesis. Topics covered included: mode awareness, industry/research contributions, human-machine interface analysis, research objectives, modeling, flight guidance project and evaluation.

Degani, Asaf↗

NASA NTRS1998

A Formal Approach for Designing and Evaluating Procedures

Operator interaction with modern control systems is a topic of great concern in high-risk industries such as nuclear power and commercial aviation. The issues associated with such systems focus on the ability of the operators (e.g., pilots) to achieve mission goals safely while containing failures. Operators must be able to interact safely and reliably with highly automatic and complex systems across the full spectrum of possible operating conditions, including normal, abnormal, and emergency situations. In environments such as commercial aviation, operator interaction with the machine is specified through a set of standard operating procedures (SOP). A procedure represents a collective agreement on the 'best' way to perform a given task. The intent of this paper is to suggest a formal methodology, for designing and evaluating procedures, that is both reliable and systematic. Our approach involves two major elements: a model of the machine and a list of the operator's task specifications (goals). We use formal modeling paradigms for describing the system and super-imposing on it the operator's tasks. Such paradigms, based on recent frameworks such as Statecharts and Hierarchical Hybrid Machines appear to be adequate methods for analyzing operator interaction with modern control systems. To illustrate this methodology, we model and analyze the sequence of actions for an emergency procedure. The procedure, Irregular Engine Start, for a medium-range aircraft, specifies the sequence of immediate actions that must be performed by the crew to avoid an uncontrolled rise in engine temperature during start-up. A model of engine behavior during a hot start is constructed. It also describes the various actions that can be taken by the crew and the resulting outcomes. The model is then opened up as a tree of all possible action sequences. This action tree allows us to trace the correct sequences necessary to achieve the desired end-goal (secure and shut down of the engine). In conclusion, we argue that the current process of designing and evaluating procedures can be improved. We discuss the implications of this approach for designing and evaluating this and other types of procedures. We conclude with insights about the benefits and limitation of this methodology, and offer suggestions for future research.

Degani, Asaf↗

NASA NTRS1998

Some Formal Aspects of Human-Machine Interaction

While automated control systems such as autopilots and medical devices are introduced at a rapid pace, it is widely recognized that user interaction with these machines is problematic (Abbott, Slotte, & Stimson, 1996). One factor commonly cited in the literature is the discrepancy between the machine's behavior and the user's expectations. Design guidelines to reduce this discrepancy focus on two elements: (1) improvement of the "feedback" about what the automation is actually doing, and (2) improvement of the user's "mental model" of the automation (Norman, 1990; Sarter and Woods, 1995). This presentation describes a methodology for investigating these two elements via a formal (Le., mathematical) approach. The method involves two representations: (1) a finite state model of the machine's behavior (2) a finite state model of the user's knowledge and expectations about the machine's behavior. In the analysis phase we compare these two models and identify discrepancies. Such discrepancies can be compensated by augmenting the display and/or the user's model. A taxonomy of these discrepancies will be discussed using examples from automated Eight control systems of modern "glass cockpit" jetliners.

Degani, Asaf↗

NASA NTRS1997

Dual-Task Interference Across Practice: Reductions in Task Two Slowing

When subjects attempt to perform two speeded discrimination tasks in in rapid succession, performance on the second discrimination is often strikingly impaired. It susc previously been reported that extended practice yields only a minimal decrease in the amount of Task 2 slowing. In contrast, we found that Task 2 slowing decreases dramatically for all six subjects across 40 sessions of practice. In one experiment we demonstrate that this reduction in interference does not occur when the tasks use the same response modality. Two additional experiments investigating transfer effects demonstrate that dual-task interference remains small when a new (unpracticed) Task 2 is used, but returns to baseline when a new Task 1 is used. We conclude that the main cause of the changes that occur with extended practice is a drastic shortening of bottleneck stages in Task 1.

VanSelst, Mark↗

NASA NTRS1997

Response Modality Modulator PRP Interference, Even When Task 1 is a no-go trial

In a Psychological Refractory Period paradigm where the first task requires one of several possible responses on some trials (the go trials), but no response on other (no-go) trials, response modality modulates the amount of interference on Task 2 processing. This is true even when Task 1 is no-go trial. Differences in the cost of switching response sets between processing for Task 1 and Task 2 may provide an account of this surprising phenomenon.

VanSelst, Mark↗

NASA NTRS1997

Word Effects in Dual-Task Studies Using Lexical Decision and Naming as Task 2

Word frequency effects in dual-task, lexical decision are variously reported to be additive or under-additive across SOA. We replicate and extend earlier lexical decision studies and find word frequency to be additive across SOA. To more directly capture lexical processing, we examine dual-task naming. Once again we find word frequency to be additive across SOA. Lexical processing appears to be constrained by central processing limitations.

Remington, Roger↗

NASA NTRS1997

Practice Drastically Reduces Interference in the Psychological Refractory Period Paradigm

Six subjects were given 36 sessions of practice in a PRP paradigm. Task 1 required vocal responses, while Task 2 required manual responses. Unlike previous experiments with two manual response tasks, practice dramatically reduced the size of the PRP interference, from an initial 353 msec to only 40 msec. Analysis of factor interactions support a central bottleneck model of Task 2 postponement both early and late in practice.

VanSelst, Mark↗

NASA NTRS1995

Mode Transitions in Glass Cockpit Aircraft: Results of a Field Study

One consequence of increased levels of automation in complex control systems is the presence of modes. A mode is a particular configuration of a control system that defines how human command inputs are interpreted. In complex systems, modes also often determine a specific allocation of control authority between the human and automated systems. Even in simple static devices (e.g., electronic watches, word processors), the presence of modes has been found to cause problems in either-the acquisition or production of skilled performance. Many of these problems arise due to the fact that the selection of a mode causes device behavior to be mediated by hidden internal state information. For these simple systems, many of these interaction problems can be solved by the design of appropriate feedback to communicate internal state information to the human operator. In complex dynamic systems, however, the design issues associated with modes seem to trancend the problem of merely communicating internal state information via displayed feedback. In complex supervisory control systems (e.g., aircraft, spacecraft, military command and control), a key function of modes is the selection of a particular configuration of control authority between the human operator and automated control systems. One mode may result in full manual control, another may result in a mix of manual and automatic control, while a third may result in full automatic control over the entire system. The human operator selects an appropriate mode as a function of current goals, operating conditions, and operating procedures. Thus, the operator is put in a position of essentially trying to control two coupled dynamic systems: the target system itself, and also a highly complex suite of automation controlling the target system. From a historical perspective, it should probably not come as a surprise that very little information is available to guide the design of mode-oriented control systems. The topic of function allocation (i.e., the proper division of control authority among human and computer) has a long history in human-machine systems research. Although this research has produced some relevant guidelines, a design approach capable of defining appropriate allocations of control function between the human and automation is not yet available. As a result, the function allocation decision itself has been allocated to the operator, to be performed in real-time, in the operation of mode-oriented control systems. A variety of documented aircraft accidents and incidents suggest that the real-time selection and monitoring of control modes is a weak link in the effective operation of complex supervisory control systems. Research in human-machine systems and human-computer interaction has barely scraped the surface of the problem of understanding how operators manage this task.The purpose of this paper is to present the results of a field study which examined how operators manage mode selection in a complex supervisory control system. Data on mode engagements using the Boeing B757/767 auto-flight system were collected during approach and descent into four major airports in the East Coast of the United States. Protocols documenting mode selection, automatic mode changes, pilot actions, quantitative records of flight-path variables, and verbal reports during and after mode engagements were collected by an observer from the jumpseat. Observations were conducted on two typical trips between three airports. Each trip was be replicated 11 times, which yielded a total of 22 trips and 66 legs on which data were collected. All data collected concerned the same flight numbers, and therefore, the same time of day, same type of aircraft, and identical operational environments (e.g., ATC facilities, weather patterns, traffic flow etc.)

Degani, Asaf↗

NASA NTRS1994

Models of Human Information Requirements: "When Reasonable Aiding Systems Disagree"

Aircraft flight management and Air Traffic Control (ATC) automation are under development to maximize the economy of flight and to increase the capacity of the terminal area airspace while maintaining levels of flight safety equal to or better than current system performance. These goals are being realized by the introduction of flight management automation aiding and operations support systems on the flight deck and by new developments of ATC aiding systems that seek to optimize scheduling of aircraft while potentially reducing required separation and accounting for weather and wake vortex turbulence. Aiding systems on both the flight deck and the ground operate through algorithmic functions on models of the aircraft and of the airspace. These models may differ from each other as a result of variations in their models of the immediate environment. The resultant flight operations or ATC commands may differ in their response requirements (e.g. different preferred descent speeds or descent initiation points). The human operators in the system must then interact with the automation to reconcile differences and resolve conflicts. We have developed a model of human performance including cognitive functions (decision-making, rule-based reasoning, procedural interruption recovery and forgetting) that supports analysis of the information requirements for resolution of flight aiding and ATC conflicts. The model represents multiple individuals in the flight crew and in ATC. The model is supported in simulation on a Silicon Graphics' workstation using Allegro Lisp. Design guidelines for aviation automation aiding systems have been developed using the model's specification of information and team procedural requirements. Empirical data on flight deck operations from full-mission flight simulation are provided to support the model's predictions. The paper describes the model, its development and implementation, the simulation test of the model predictions, and the empirical validation process. The model and its supporting data provide a generalizable tool that is being expanded to include air/ground compatibility and ATC crew interactions in air traffic management.

Corker, Kevin↗