Search NASA⌕ Search

Engineering topics

Holloway, C. M.

Publications and source records attributed to Holloway, C. M..

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2015

"Evidence" Under a Magnifying Glass: Thoughts on Safety Argument Epistemology

Common definitions of "safety case" emphasize that evidence is the basis of a safety argument, yet few widely referenced works explicitly define "evidence". Their examples suggest that similar things can be regarded as evidence. But the category evidence seems to contain (1) processes for finding things out, (2) information resulting from such processes, and (3) relevant documents. Moreover, any item of evidence could be replaced by further argument. Normative models of informal argumentation do not offer clear guidance on when a safety argument should cite evidence rather than appeal to a more detailed argument. Disciplines such as the law address the problem with a practical, domain-specific epistemology. In this paper, we explore these problems associated with evidence citations in safety arguments, identify goals for a theory of safety argument evidence and a practical safety argument epistemology, propose a model of safety evidence citation that advances the identified goals, and present a related extension to the Goal Structuring Notation (GSN).

Graydon, P. J.↗
NASA NTRS2015

Planning the Unplanned Experiment: Towards Assessing the Efficacy of Standards for Safety-Critical Software

Safe use of software in safety-critical applications requires well-founded means of determining whether software is fit for such use. While software in industries such as aviation has a good safety record, little is known about whether standards for software in safety-critical applications 'work' (or even what that means). It is often (implicitly) argued that software is fit for safety-critical use because it conforms to an appropriate standard. Without knowing whether a standard works, such reliance is an experiment; without carefully collecting assessment data, that experiment is unplanned. To help plan the experiment, we organized a workshop to develop practical ideas for assessing software safety standards. In this paper, we relate and elaborate on the workshop discussion, which revealed subtle but important study design considerations and practical barriers to collecting appropriate historical data and recruiting appropriate experimental subjects. We discuss assessing standards as written and as applied, several candidate definitions for what it means for a standard to 'work,' and key assessment strategies and study techniques and the pros and cons of each. Finally, we conclude with thoughts about the kinds of research that will be required and how academia, industry, and regulators might collaborate to overcome the noted barriers.

Graydon, Patrick J.↗
NASA NTRS2012

Comparing the Identification of Recommendations by Different Accident Investigators Using a Common Methodology

Accident reports play a key role in the safety of complex systems. These reports present the recommendations that are intended to help avoid any recurrence of past failures. However, the value of these findings depends upon the causal analysis that helps to identify the reasons why an accident occurred. Various techniques have been developed to help investigators distinguish root causes from contributory factors and contextual information. This paper presents the results from a study into the individual differences that can arise when a group of investigators independently apply the same technique to identify the causes of an accident. This work is important if we are to increase the consistency and coherence of investigations following major accidents.

Johnson, Chris W.↗
NASA NTRS2010

Safety Arguments for Next Generation, Location Aware Computing

Concerns over accuracy, availability, integrity, and continuity have limited the integration of Global Positioning System (GPS) and Global Navigation Satellite System (GLONASS) for safety-critical applications. More recent augmentation systems, such as the European Geostationary Navigation Overlay Service (EGNOS) and the North American Wide Area Augmentation System (WAAS) have begun to address these concerns. Augmentation architectures build on the existing GPS/GLONASS infrastructures to support location based services in Safety of Life (SoL) applications. Much of the technical development has been directed by air traffic management requirements, in anticipation of the more extensive support to be offered by GPS III and Galileo. WAAS has already been approved to provide vertical guidance for aviation applications. During the next twelve months, the full certification of EGNOS for SoL applications is expected. This paper discusses similarities and differences between the safety assessment techniques used in Europe and North America.

Johnson, C. W.↗
NASA NTRS2010

A Safety Conundrum Illustrated: Logic, Mathematics, and Science Are Not Enough

In an ideal world, conversations about whether a particular system is safe, or whether a particular method or tool enhances safety, would be emotion-free discussions concentrating on the level of safety required, available evidence, and coherent logical, mathematical, or scientific arguments based on that evidence. In the real world, discussions about safety are often not emotion-free. Political and economic arguments may play a bigger role than logical, mathematical, and scientific arguments, and psychological factors may be as important, or even more important, than purely technical factors. This paper illustrates the conundrum that can result from this clash of the ideal and the real by means of an imagined conversation among a collection of fictional characters representing various types of people who may be participating in a safety discussion.

Holloway, C. M.↗
NASA NTRS2010

Epistemic Questions and Answers for Software System Safety

System safety is primarily concerned with epistemic questions, that is, questions concerning knowledge and the degree of confidence that can be placed in that knowledge. For systems with which human experience is long, such as roads, bridges, and mechanical devices, knowledge about what is required to make the systems safe is deep and detailed. High confidence can be placed in the validity of that knowledge. For other systems, however, with which human experience is comparatively short, such as those that rely in part or in whole on software, knowledge about what is required to ensure safety tends to be shallow and general. The confidence that can be placed in the validity of that knowledge is consequently low. In a previous paper, we enumerated a collection of foundational epistemic questions concerning software system safety. In this paper, we review and refine the questions, discuss some difficulties that attend to answering the questions today, and speculate on possible research to improve the situation.

Holloway, C. M.↗
NASA NTRS2009

Towards A Comprehensive Consideration of Epistemic Questions in Software System Safety

For any software system upon which lives depend, the most important question one can ask about it is, 'How do we know the system is safe?' Despite the critical importance of this question, no widely accepted, generally applicable answer exists. Instead, debate continues to rage over the question, with theorists and practitioners quarrelling with each other and amongst themselves. This paper suggests a possible way forward towards quelling the quarrels, based on refining the critical safety question into additional questions, which may be more likely to have answers on which a consensus can be reached.

Holloway, C. M.↗
NASA NTRS2008

How Past Loss of Control Accidents May Inform Safety Cases for Advanced Control Systems on Commercial Aircraft

This paper describes five loss of control accidents involving commercial aircraft, and derives from those accidents three principles to consider when developing a potential safety case for an advanced flight control system for commercial aircraft. One, among the foundational evidence needed to support a safety case is the availability to the control system of accurate and timely information about the status and health of relevant systems and components. Two, an essential argument to be sustained in the safety case is that pilots are provided with adequate information about the control system to enable them to understand the capabilities that it provides. Three, another essential argument is that the advanced control system will not perform less safely than a good pilot.

Holloway, C. M.↗
NASA NTRS2007

The Dangers of Failure Masking in Fault-Tolerant Software: Aspects of a Recent In-Flight Upset Event

On 1 August 2005, a Boeing Company 777-200 aircraft, operating on an international passenger flight from Australia to Malaysia, was involved in a significant upset event while flying on autopilot. The Australian Transport Safety Bureau's investigation into the event discovered that an anomaly existed in the component software hierarchy that allowed inputs from a known faulty accelerometer to be processed by the air data inertial reference unit (ADIRU) and used by the primary flight computer, autopilot and other aircraft systems. This anomaly had existed in original ADIRU software, and had not been detected in the testing and certification process for the unit. This paper describes the software aspects of the incident in detail, and suggests possible implications concerning complex, safety-critical, fault-tolerant software.

Johnson, C. W.↗
NASA NTRS2007

A Look at Aircraft Accident Analysis in the Early Days: Do Early 20th Century Accident Investigation Techniques Have Any Lessons for Today?

In the early years of powered flight, the National Advisory Committee on Aeronautics in the United States produced three reports describing a method of analysis of aircraft accidents. The first report was published in 1928; the second, which was a revision of the first, was published in 1930; and the third, which was a revision and update of the second, was published in 1936. This paper describes the contents of these reports, and compares the method of analysis proposed therein to the methods used today.

Holloway, C. M.↗
NASA NTRS2006

Why System Safety Professionals Should Read Accident Reports

System safety professionals, both researchers and practitioners, who regularly read accident reports reap important benefits. These benefits include an improved ability to separate myths from reality, including both myths about specific accidents and ones concerning accidents in general; an increased understanding of the consequences of unlikely events, which can help inform future designs; a greater recognition of the limits of mathematical models; and guidance on potentially relevant research directions that may contribute to safety improvements in future systems.

Holloway, C. M.↗
NASA NTRS2006

Questioning the Role of Requirements Engineering in the Causes of Safety-Critical Software Failures

Many software failures stem from inadequate requirements engineering. This view has been supported both by detailed accident investigations and by a number of empirical studies; however, such investigations can be misleading. It is often difficult to distinguish between failures in requirements engineering and problems elsewhere in the software development lifecycle. Further pitfalls arise from the assumption that inadequate requirements engineering is a cause of all software related accidents for which the system fails to meet its requirements. This paper identifies some of the problems that have arisen from an undue focus on the role of requirements engineering in the causes of major accidents. The intention is to provoke further debate within the emerging field of forensic software engineering.

Johnson, C. W.↗
NASA NTRS2005

A Technique for Showing Causal Arguments in Accident Reports

In the prototypical accident report, specific findings, particularly those related to causes and contributing factors, are usually written out explicitly and clearly. Also, the evidence upon which these findings are based is typically explained in detail. Often lacking, however, is any explicit discussion, description, or depiction of the arguments that connect the findings and the evidence. That is, the reports do not make clear why the investigators believe that the specific evidence they found necessarily leads to the particular findings they enumerated. This paper shows how graphical techniques can be used to depict relevant arguments supporting alternate positions on the causes of a complex road-traffic accident.

Holloway, C. M.↗
NASA NTRS2004

Distribution of Causes in Selected US Aviation Accident Reports Between 1996 and 2003

This paper describes the results of an independent analysis of the probable and contributory causes of selected aviation accidents in the United States between 1996 and 2003. The purpose of the study was to assess the comparative frequency of a variety of causal factors in the reporting of these adverse events. Although our results show that more of these high consequence accidents were attributed to human error than to any other single factor, a large number of reports also mentioned wider systemic issues, including the managerial and regulatory context of aviation operations. These wider issues are more likely to appear as contributory rather than primary causes in this set of accident reports.

Holloway, C. M.↗
NASA NTRS2004

'Systemic Failures' and 'Human Error' in Canadian TSB Aviation Reports Between 1996 and 2002

This paper describes the results of an independent analysis of the primary and contributory causes of aviation accidents in Canada between 1996 and 2003. The purpose of the study was to assess the comparative frequency of a range of causal factors in the reporting of these adverse events. Our results suggest that the majority of these high consequence accidents were attributed to human error. A large number of reports also mentioned wider systemic issues, including the managerial and regulatory context of aviation operations. These issues are more likely to appear as contributory rather than primary causes in this set of accident reports.

Holloway, C. M.↗
NASA NTRS2003

A Survey of Logic Formalisms to Support Mishap Analysis

Mishap investigations provide important information about adverse events and near miss incidents. They are intended to help avoid any recurrence of previous failures. Over time, they can also yield statistical information about incident frequencies that helps to detect patterns of failure and can validate risk assessments. However, the increasing complexity of many safety critical systems is posing new challenges for mishap analysis. Similarly, the recognition that many failures have complex, systemic causes has helped to widen the scope of many mishap investigations. These two factors have combined to pose new challenges for the analysis of adverse events. A new generation of formal and semi-formal techniques have been proposed to help investigators address these problems. We introduce the term mishap logics to collectively describe these notations that might be applied to support the analysis of mishaps. The proponents of these notations have argued that they can be used to formally prove that certain events created the necessary and sufficient causes for a mishap to occur. These proofs can be used to reduce the bias that is often perceived to effect the interpretation of adverse events. Others have argued that one cannot use logic formalisms to prove causes in the same way that one might prove propositions or theorems. Such mechanisms cannot accurately capture the wealth of inductive, deductive and statistical forms of inference that investigators must use in their analysis of adverse events. This paper provides an overview of these mishap logics. It also identifies several additional classes of logic that might also be used to support mishap analysis.

Johnson, Chris↗
NASA NTRS2002

The Strengths and Weaknesses of Logic Formalisms to Support Mishap Analysis

The increasing complexity of many safety critical systems poses new problems for mishap analysis. Techniques developed in the sixties and seventies cannot easily scale-up to analyze incidents involving tightly integrated software and hardware components. Similarly, the realization that many failures have systemic causes has widened the scope of many mishap investigations. Organizations, including NASA and the NTSB, have responded by starting research and training initiatives to ensure that their personnel are well equipped to meet these challenges. One strand of research has identified a range of mathematically based techniques that can be used to reason about the causes of complex, adverse events. The proponents of these techniques have argued that they can be used to formally prove that certain events created the necessary and sufficient causes for a mishap to occur. Mathematical proofs can reduce the bias that is often perceived to effect the interpretation of adverse events. Others have opposed the introduction of these techniques by identifying social and political aspects to incident investigation that cannot easily be reconciled with a logic-based approach. Traditional theorem proving mechanisms cannot accurately capture the wealth of inductive, deductive and statistical forms of inference that investigators routinely use in their analysis of adverse events. This paper summarizes some of the benefits that logics provide, describes their weaknesses, and proposes a number of directions for future research.

Johnson, C. W.↗