Engineering topics

Hannah Walsh

Publications and source records attributed to Hannah Walsh.

17 records

NASA NTRS2024

Workshop on the Role of Design Assurance in System-Wide Safety’s Safety Demonstrator Series

This report summarizes a three-hour hybrid in-person/online workshop on June 7, 2024 on the topic of design assurance and the Safety Demonstrator Series (SDS), which was held at NASA Ames Research Center. The SDS provides an operational demonstration of, and recommendations for, requirements and standards necessary to monitor, assess, and mitigate risks to assure safety in disaster-oriented operations. Over sixty NASA personnel participated in the workshop. Four main topics were discussed: (1) assurance needs for the Safety Demonstrator Series, (2) assurance and the In-Time Aviation Safety Management System (IASMS), (3) in-time assurance: existing efforts and future opportunities, and (4) demonstrating assurance tools in the Safety Demonstrators. Key takeaways are as follows: 1. Design assurance tools can be used to assure an In-Time Aviation Safety Management System, the systems that comprise it, and other systems or missions. Assurance must consider both systems and components and include the interactions between elements in both a systems/aircraft context and a systems-of-systems/airspace context. 2. Design-time assurance activities can support the identification of monitors needed for operational assurance activities (i.e., the “monitor” function in the monitor-assess-mitigate paradigm at the heart of the IASMS concept). 3. A major opportunity for design-time assurance tools to contribute to the IASMS concept is to support rapid re-validation of systems. This will be particularly important for (1) supporting novel operations in the IASMS, where operational data may disprove design-time assumptions (motivating re-analysis of system safety) and (2) adapting technologies (e.g., AI/ML for autonomous operations) to new operational domains, where there may be new or different safety considerations not included in the initial scope of operations. 4. There are several design assurance tools under development in the System-Wide Safety project that can support assurance of Services, Functions, and Capabilities (SFCs) in the Safety Demonstrators. Transitioning these tools from one-off research projects into a functioning part of IASMS assurance will require closer integration between these tools. 5. It is not clear whether the role of design assurance tools is primarily as a part of IASMS architecture or as an external check on IASMS. The workshop consisted of four discussion topics initiated via four lightning talks by System-Wide Safety researchers. Discussions utilized Mural to engage both in-person and online participants in the hybrid format. Polls and surveys were also utilized to gather participant input. The workshop closed with a reflection activity for participants, as well as new ideas for collaboration and coordination of ongoing System-Wide Safety research.

design assurance↗

NASA NTRS2023

Wildfire-fighting Use Case Requirements to Monitor

In this technical report, we provide requirements for a wildfire-fighting use-case, towards the Safety Demonstrator 1. The use case will incorporate ground and airborne assets operating in a coordinated fashion, and will comprise five activities, from detection to the execution of the initial attack. Depending on the activity and the data involved, the requirements identified may be non-probabilistic or probabilistic. In both cases, we first identify some of the requirements we wish to monitor, and then present a formalization using the language of requirements of the NASA requirements elicitation tool FRET. To formalize probabilistic requirements, we use a novel extension to FRET’s requirements language that incorporates notions of probability, and discuss how requirements can be translated into existing probabilistic temporal logics like PCTL. We exemplify how some of the requirements presented can be monitored using the existing tools Ogma and Copilot. We close with a summary and future directions.

Requirements↗

NASA NTRS2022

Natural Language Processing Techniques for Intelligent Knowledge Management of Safety Reports

Safety, failure, and incident reports are common artifacts across various domains, including aviation and wildfire response. These reports are often mandatory to submit, resulting in the culmination of large repositories of text-based documents. Simultaneously, these reports and corresponding repositories are often only manually analyzed and queried by users via out-of-date search engines. As a consequence, we have been developing the Manager for Intelligent Knowledge Access (MIKA) toolkit, which uses natural language processing to improve information access and reuse. In this presentation, we discuss natural language processing techniques for knowledge discovery and apply these methods to a repository of aerial wildfire mishap reports. Two methods are used for knowledge discovery: topic modeling and named-entity recognition. We use topic modeling to identify hazards and perform a trend analysis to produce a data-driven risk matrix. A custom named-entity recognition model, build from fine tuning a pre-trained language model, is used to identify failure modes, failure causes, failure effects, control processes, and recommendations to aid in failure modes and effects analysis (FMEA). Throughout the presentation, we discuss and apply natural language processing techniques to better leverage the vast amount of information contained in report repositories.

Machine learning↗

NASA NTRS2022

NASA System-Wide Safety Wildland Firefighting Operations Workshop Report

On March 9-11, 2022, NASA’s System-Wide Safety Wildland Firefighting Operations Workshop engaged the broader wildland firefighting management ecosystem in a safety-oriented discussion via a virtual platform. This enabled a better understanding of how NASA and community expertise can be leveraged in the safe development of current and future firefighting systems and operations. The goals of the workshop were to: (1) identify and prioritize the top safety-oriented risks, gaps in capabilities, and emerging technologies to enhance wildland firefighting for both near-term and far-term concepts, with a specific focus on aviation operations and (2) engage the stakeholder community in defining emergent safety-oriented scope, roles, responsibilities, and procedures for agents undergoing increasingly complex wildland firefighting operations in information-rich, but uncertain environments. Workshop participants were solicited from wildland firefighting stakeholders across government, industry, and academia. All levels of government were engaged, as NASA sought attendees from federal, state, local, and tribal government agencies. Industry participants from traditional wildland firefighting domains such as data visualization and equipment manufacturers were invited, and corporate attendees from novel application domains such as aerial robotics and autonomous systems were present as well. The top three findings were as follows: (1) Enhancing situation awareness is a safety priority, especially in the use of aerial assets; (2) Timely access to information along with data fusion and integrated displays will enhance safety-critical decision-making both inside and outside aviation contexts; and (3) Tailorable standards and common operating pictures in the field will enhance inter-agency cooperation in the wildland firefighting lifecycle and enable the optimal use of limited resources such as aerial assets. The workshop helped inform NASA of the relevant safety-related wildland firefighting concerns and aided the broader ecosystem in understanding the potential safety-oriented role NASA might play in this community. Increased engagement with crucial governmental stakeholders (e.g., U.S. Forest Service, CAL FIRE, etc.) along with industry partners in cutting- edge information -centric domains is a fundamental next step. Additionally, the workshop findings will help define the first of a series of operationally challenging demonstrations, held in concert with strategic ecosystem partners, known as the Safety Demonstrator Series for NASA’s System-Wide Safety project. The first demonstration is set in the wildland firefighting application domain and will: (1) examine high risk operational scenarios to reduce their overall risk via services, functions or capabilities that act as risk mitigators (or transfer that risk to automated systems better able to tolerate it) and (2) explore novel tools and technologies that will enhance safety margins by enabling non-traditional or neoteric operational paradigms.

wildland firefighting↗

NASA NTRSDate not supplied

The System Modeling and Analysis of Resiliency in STEReO (SMARt-STEReO)

Wildfire emergency response has remained rooted in relatively low-tech solutions for coordination between ground and aerial assets. These low-tech solutions are robust for the remote environments in which wildfires are usually fought, but limit strategic cross-organizational support and the ability to deploy and effectively utilize aerial assets. As aircraft become more advanced and new technology, including drones, become available to firefighters, a new, more modern method of asset coordination is needed. NASA is working on a project called ‘Scalable Traffic Management for Emergency Response Operations’ (STEReO) to integrate unmanned aerial systems (UAS)and UAS traffic management (UTM)into wildfire response. STEReO’s goals include simplifying the coordination of aerial assets, improving the existing UAS framework, and increasing the role of additional autonomous systems to reduce human risk and to increase system resilience. This paper describes the development of the ‘System Modeling and Analysis of Resiliency in STEReO’ (SMARt-STEReO) project, which aims to model wildfire response and to quantify the additional system resilience that STEReO technology provides firefighters. This paper verifies SMARt-STEReO and defines its scope; it includes experimental and statistical analysis of the impact that the addition of UAS has on both performance metrics and also on performance resiliency response to a given fault. SMARt-STEReO is a grid-based model of fire propagation that incorporates varying crew responses. Through the use of a Python package called ‘fmdtools’, the model easily allows for the addition of faults to the system. These faults allow analysts to investigate various response parameters. Factors including terrain, fuel type and wind speed can be modified to affect the fire propagation; additionally, the number of ground crews, engines, fixed wing aircraft, helicopters, and UAS can be changed to affect the crew response. The communication lines between actors mimic those used in real life situations. This paper explains the development of SMARt-STEReO including background research, verification and validation, and preliminary experimental analysis of system resilience to both a minor and major fault in systems with and without UAS.

Resiliency↗

NASA NTRSDate not supplied

The System Modeling and Analysis of Resiliency in STEReO (SMARt-STEReO)

NASA's Scalable Traffic Management for Emergency Response Operations (STEReO) project aims to leverage Unmanned Aerial Systems (UAS) and UAS Traffic Management (UTM) to improve asset coordination and overall emergency response. One application of STEReO is wildfire response, which is the focus of this research. In order to implement the operations described in the STEReO project, these additions must have tangible benefits and proven safety. To this end, the System Modeling and Analysis of Resiliency in STEReO (SMARt-STEReO) project constructs a simulation model, developed through the Python modeling and resiliency analysis package fmdtools. The model describes wildfire response operations, including current operational concepts and emerging concepts utilizing UAS as described in STEReO. While previous simulation models focus primarily on fire propagation with some models including emergency response intervention, SMART-STEReO evaluates the system performance and resilience benefits gained by the addition of UAS and UTM. Due to the novelty and complexity of the model, initial model verification and validation efforts are conducted and a detailed description of the model is provided. Preliminary results from experimental analysis on the SMARt-STEReO model indicate that when compared to current operations, the addition of UAS in wildfire operations results in improved response efforts, in terms of fewer acres burned, as well as improved system resilience in response to a given fault.

Sequoia Andrade↗

NASA NTRSDate not supplied

Machine Learning Enabled Quantitative Risk Assessment of Aerial Wildfire Response

Aerial wildfire operations are high risk and account for a large number of firefighter deaths. Increasing intensity of wildfires is driving a surge in aerial operations, while simultaneously there is growing interest in improving system safety and performance. In this work, wildfire aviation mishaps documented using the SAFECOM system are analyzed using a previously developed framework for hazard extraction and analysis of trends (HEAT). Hazards and specific failure modes are extracted from the narrative data in SAFECOM forms using natural language processing techniques. Metrics for each hazard are calculated, including frequency, rate, and severity. We examine whether these metrics change over time, and whether they are related to metadata, such as region and aircraft type. The results of the hazard analysis are presented in a risk matrix, identifying the highest and lowest risk hazards based on rate of occurrence and average severity. Results identify jumper operations hazards as high-risk, in addition to bucket drop failures, cargo let down failures, and severe weather as medium risk.

machine learning↗

NASA NTRSDate not supplied

What Went Wrong: A Survey of Wildfire UAS Mishaps through Named Entity Recognition

Increasingly, unmanned aircraft systems (UAS) are being applied to wildfire incidents for tasks such as mapping, aerial ignition, and delivery. As a result, incident reporting systems for wildfires are beginning to accumulate data related to UAS mishaps in wildfire response. In this research, we apply state-of-the-art natural language processing (NLP) techniques to develop a custom Named Entity Recognition (NER) model which extracts a Failure Modes and Effects Analysis (FMEA)-style survey of wildfire UAS mishaps reported in SAFECOM. The custom NER model is built by fine-tuning an existing (BERT) model, resulting in a generalizable NER model that can extract engineering relevant entities including failure modes, causes, effects, control processes, and recommendations from any failure-relevant text. Similar mishaps are clustered and reported as single rows within the FMEA. For each cluster, frequency, severity, and overall risk are computed. The methodology can be applied as part of a broader safety management system to track trends in mishaps and discover knowledge that can be utilized to improve safety outcomes and system performance.

Machine Learning↗

NASA NTRSDate not supplied

MIKA: Manager for Intelligent Knowledge Access Toolkit for Engineering Knowledge Discovery and Information Retrieval

Repositories of safety reports are often underutilized and only analyzed manually by trained experts, despite safety management systems requiring reports. These collections of documents contain a wealth of information from past projects and operations that could improve system safety and design. Advances in natural language processing techniques have improved information extraction and retrieval in consumer technology, biomedicine, and finance, for instance, but have not been applied to engineering documents on the same scale. To this end, the Manager for Intelligent Knowledge Access (MIKA) open-source toolkit has been developed for rapid knowledge discovery and information retrieval in safety engineering applications. The MIKA toolkit uses state-of-the-art natural language processing algorithms and allows a user to apply these methods to their own dataset. This paper describes the MIKA toolkit and its two primary capabilities, knowledge discovery and information retrieval, and demonstrates the toolkit via a case study on National Transportation Safety Board (NTSB) reports.

Machine Learning↗

NASA NTRSDate not supplied

Open Data Integration (ODIN): A Concurrent, Distributed Message-Based Architecture and Framework for Disaster Response

The Runtime for Airspace Concept Evaluation (RACE) is an open-source software architecture and framework to build configurable, highly concurrent and distributed message-based systems that offer scalable, low-latency performance on commodity hardware. RACE was used in commercial aviation applications to rapidly build systems that span several machines (including synchronized displays), interface existing hardware simulators and other live data feeds, and incorporate sophisticated visualization components such as NASA WorldWind. These RACE applications validated elements of the FAA’s System Wide Information Management (SWIM) Program, handling up to 1000 messages/sec from diverse sources (SFDPS, TFM-DATA, TAIS, ASDE-X, ITWS and local ADS) for 4,500 simultaneous flights tracked in the next-generation air transportation system’s digital backbone. We have since generalized RACE to support Open Data Integration (ODIN) applications outside aviation. Systems built with RACE/ODIN can be deployed in the field, on commodity hardware, and operate with limited or intermittent connectivity to the outside world. Our primary use case is a web-server with local/persistent data storage that runs within and only serves the stakeholder network (e.g. an incident command post). We are tailoring the RACE/ODIN system to support wildland fire management for the upcoming NASA Wildland Fire Safety Demonstration Series. RACE-ODIN is under consideration for application in the Scalable Traffic Management for Emergency Response Operations project, or STEReO, which aims to create a system that can be deployed during emergencies, to coordinate multiple elements of disaster response. Such data sources predominantly come from existing services on the internet (e.g. weather and satellite data, imported from so called "edge servers") but can also include dynamic (real-time) data from computer simulations and within the stakeholder network (such as aircraft and personnel tracking information). We will present the architecture and ODIN system demonstration incorporating local data from instrumented power-line towers, interpolated weather data and geospatial data from space-based platforms.

Guillaume P Brat↗

NASA NTRSDate not supplied

MIKA: Manager for Intelligent Knowledge Access Toolkit for Engineering Knowledge Discovery and Information Retrieval

Systems Engineering↗

NASA NTRSDate not supplied

SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model

As aviation systems continue to operate with high traffic, large amounts of documents containing safety-relevant data continue to be generated via reporting systems such as the Aviation Safety Reporting System (ASRS). Advanced natural language processing techniques, specifically pre-trained language models, have shown great success in domain-specific applications; however, the text in aviation safety reports is inundated with jargon and thus not fully utilized by general pre-trained models. In this research, we work towards developing a safety-informed aerospace-specific language model by pre-training a Bidirectional Encoder Representations from Transformer (BERT) model on reports from the Aviation Safety Reporting System and the National Transportation Safety Board. The resulting model, called SafeAeroBERT, is fine-tuned for the specific task of document classification, and can be further tuned for named-entity recognition, relation detection, information retrieval, and summarization. Results from the classification task are compared between SafeAeroBERT, the base BERT, and SciBERT models and show SafeAeroBERT outperforms the general BERT and SciBERT on classifying reports about weather and procedure. SafeAeroBERT can be used on custom tasks, not limited to document classification, and is intended to aid an intelligent knowledge manager for safety report repositories.

Aviation↗

NASA NTRSDate not supplied

Detecting and Characterizing Patterns of Failure in Complex Systems: An Ontology Development and Clustering Approach

While the causes of failures in complex engineered systems are often clear in hindsight, it can be challenging to predict failures proactively during the design of novel engineered products or systems. Identifying patterns can be useful for capturing common characteristics that may lead to failure. In this paper, we present a methodology for identifying patterns of failure from NASA’s publicly available Lessons Learned Information System (LLIS). We apply an ontology development and clustering approach to identify representative patterns leading to failures in historical lessons learned. A joint inductive-deductive approach reveals the key themes in lessons that lead to failure, which are formalized and recorded as an ontology of complex systems failure causes. Documents from the LLIS are manually tagged with relevant characteristics from the ontology. From the tagged set, clustering is used to capture co-occurring sets of characteristics that lead to failure. The primary contribution of this work is a method for extracting a set of generic failure patterns in complex engineered systems and characteristics for these patterns that can be identified at design time, knowledge of which can be used to plan mitigation strategies.

Systems Engineering↗

NASA NTRSDate not supplied

BERT-based Topic Modeling and Information Retrieval to Support Fishbone Diagramming for Safe Integration of Unmanned Aircraft Systems in Wildfire Response

Recent concepts for emerging wildfire response operations have included unmanned aircraft systems (UAS) due to their increasing accessibility and capabilities. To integrate UAS into wildfire response safely, researchers have studied the use of large repositories of historic incident reports to improve the scope of root cause analysis. Recent work has emphasized applying state-of-the-art natural language processing techniques to extract useful information from these repositories. However, it has not yet been studied how these results can be interpreted and integrated into the systems engineering process. In this work, we propose a process in which Bidirectional Encoder Representations from Transformers (BERT)-based topic modeling and information retrieval are applied to a relevant set of documents in order to support the development of a fishbone diagram in a semiautomated process. High-level themes in the document set are identified using topic modeling, which are then refined and interpreted by a human analyst. Then, the themes are used to guide a finer search using information retrieval, which returns specific incident reports of relevance. This provides traceability to specific incidents as well as broader categorizations that comprise the fishbone branches. We apply the proposed process to relevant documents from NASA’s Aviation Safety Reporting System (ASRS). The proposed process is widely applicable when relevant documents are available, and the results from this study will be useful to identifying potential causes of wildfire response UAS incidents.

hazard analysis↗

NASA NTRSDate not supplied

BERT-based Topic Modeling and Information Retrieval to Support Fishbone Diagramming for Safe Integration of Unmanned Aircraft Systems in Wildfire Response

hazard analysis↗

NASA NTRSDate not supplied

Identifying Human Errors and Error Mechanisms From Accident Reports Using Large Language Models

Emerging operational concepts for aviation hinge on novel paradigms for human machine interaction. Critical to their safe operation is early consideration of human error into the design process. Existing methods for consideration of human error require significant expert input, which is challenging both in early design and in novel systems for which there is little existing safety expertise. In this research, we propose a methodology for identifying human error, error producing factors, and mechanisms in early design from historical incident reports. Additionally, we hypothesize that cross-domain sharing of lessons learned can aid with early design human considerations in circumstances where data is not relevant or incomplete. This is addressed by identifying causes of human error in aviation and railway domains through applying state-of-the art natural language processing techniques to historical incident reports. Using this method, it is possible to extract extensive reports on human error from past incidents. Using the proposed approach, we identify nine human errors from railway reports and fourteen from aviation reports, with three errors common to both domains. There is at least one error producing conditions for each human error while a majority of the errors have more than one error mechanism. We also found that a majority of the human errors, error producing factors, and error mechanisms (even if they are not common between the domains) can be used to inform safe operations across domains as long as the errors are not domain specific and are interpreted and contextualized using engineering judgement.

Human Errors↗

NASA NTRSDate not supplied

Towards Functional Hazard Assessment (CFHA): A Gap Analysis and Concept for Emerging Aviation Systems

Given the current evolution of the National Airspace and future trajectory towards novel and evolving operations with varying levels of autonomy, complexity, and acceptable risk, there is an opportunity to support safety assurance by extending existing methodologies, such as Functional Hazard Assessment (FHA). In response to challenges in performing FHA for novel aviation concepts, we propose a concept for Computational Functional Hazard Assessment (CFHA), which provides processes, methods, and tools for incorporating external data to facilitate further exploration of the hazard space iterativelty throughout the design process. The core components of CFHA involve knowledge capture from historical and operational data, functional architecture specification via a formal modeling language, and simulation for hazardous scenario analysis. Through this concept, we aim to adapt conventional safety assessment to address the increasingly complex hazard space generated from emerging operations.

Seydou Mbaye↗