Results for “Decision Making, Organizational” — Page 4

NASA NTRS2020

Urban Air Mobility: Predictive Modelling on the Behavior of Air-Service Bookings in Metropolitan Areas

The demand for a ride to work without running into the inefficiencies of traffic in a bustling, metropolitan area beckons for a system to tackle on-the ground traffic congestion, thus fueling the need for Urban Air Mobility (UAM). The instant gratification culture, notably its effect on Generation Z, has given way to the nature of impulsive decision making. The 2020 COVID-19 crisis, with its use of tap-to-gratify technology, highlights passengers preferring late booking. This last-minute airport reorganization crisis spearheaded our research goal to create a more user-personalized assessment of cancellation rates. Our research highlights that there will be a market in the 2030s for UAM, short-distance air transportation, and the world must prepare for the challenges that come with this new market. Through a two-part model, we obtained two randomized sets of profiles of user-specific and ridership attributes and their relationship with cancellations. Recommendations for future research include factoring this personalized aspect into cancellation projections to provide reasoned user discounts as well as a remedy for the scheduling chaos that stems from the airport organizational crisis.

Srushti Adesara↗

NASA NTRSDate not supplied

Algorithmic Analysis of the Effect of Human Behavior on Future Urban Air Mobility Operations

The demand for a ride to work without running into the inefficiencies of traffic in a bustling, metropolitan area beckons for a system to tackle on-the ground traffic congestion, thus fueling the need for Urban Air Mobility (UAM). The instant gratification culture, notably its effect on Generation Z, has given way to the nature of impulsive decision making. The 2020 COVID-19 crisis, with its use of tap-to-gratify technology, highlights passengers preferring late booking. This last-minute airport reorganization crisis spearheaded our research goal to create a more user-personalized assessment of cancellation rates. Our research highlights that there will be a market in the 2030s for UAM, short-distance air transportation, and the world must prepare for the challenges that come with this new market. Through a two-part model, we obtained two randomized sets of profiles of user-specific and ridership attributes and their relationship with cancellations. Recommendations for future research include factoring this personalized aspect into cancellation projections to provide reasoned user discounts as well as a remedy for the scheduling chaos that stems from the airport organizational crisis.

Harbani Jaggi↗

NASA NTRS1998

Errors in Aviation Decision Making: Bad Decisions or Bad Luck?

Despite efforts to design systems and procedures to support 'correct' and safe operations in aviation, errors in human judgment still occur and contribute to accidents. In this paper we examine how an NDM (naturalistic decision making) approach might help us to understand the role of decision processes in negative outcomes. Our strategy was to examine a collection of identified decision errors through the lens of an aviation decision process model and to search for common patterns. The second, and more difficult, task was to determine what might account for those patterns. The corpus we analyzed consisted of tactical decision errors identified by the NTSB (National Transportation Safety Board) from a set of accidents in which crew behavior contributed to the accident. A common pattern emerged: about three quarters of the errors represented plan-continuation errors, that is, a decision to continue with the original plan despite cues that suggested changing the course of action. Features in the context that might contribute to these errors were identified: (a) ambiguous dynamic conditions and (b) organizational and socially-induced goal conflicts. We hypothesize that 'errors' are mediated by underestimation of risk and failure to analyze the potential consequences of continuing with the initial plan. Stressors may further contribute to these effects. Suggestions for improving performance in these error-inducing contexts are discussed.

Orasanu, Judith↗

NASA NTRS2007

Emergency Preparedness for Catastrophic Events at Small and Medium Sized Airports: Lacking or Not?

The implementation of security methods and processes in general has had a decisive impact on the aviation industry. However, efforts to effectively coordinate varied aspects of security protocols between agencies and general aviation components have not been adequately addressed. Whether or not overall security issues, especially with regard to planning for catastrophic terrorist events, have been neglected at the nation's smaller airports is the main topic of this paper. For perspective, the term general aviation is generally accepted to include all flying except for military and scheduled airline operations. Genera aviation makes up more than 1 percent of the U.S. Gross Domestic Product and supports almost 1.3 mission high-skilled jobs in professional services and manufacturing and hence is an important component of the aviation industry (AOPA, n.d.). In both conceptual and practical terms, this paper argues for the proactive management of security planning and repeated security awareness training from both an individual and an organizational perspective within the general aviation venue. The results of a research project incorporating survey data from general aviation and small commercial airport managers as well as Transportation Security Administration (TSA) employees are reported. Survey findings suggest that miscommunication does take place on different organizational levels and that between TSA employees and airport management interaction can be contentious and cooperation diminished. The importance of organizational training for decreasing conflict and increasing security and preparedness is discussed as a primary implication.

Sweet, Kathleen M.↗

NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek A Whitman↗

NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek Whitman↗

NASA NTRS2002

Making Sense of Rocket Science - Building NASA's Knowledge Management Program

The National Aeronautics and Space Administration (NASA) has launched a range of KM activities-from deploying intelligent "know-bots" across millions of electronic sources to ensuring tacit knowledge is transferred across generations. The strategy and implementation focuses on managing NASA's wealth of explicit knowledge, enabling remote collaboration for international teams, and enhancing capture of the key knowledge of the workforce. An in-depth view of the work being done at the Jet Propulsion Laboratory (JPL) shows the integration of academic studies and practical applications to architect, develop, and deploy KM systems in the areas of document management, electronic archives, information lifecycles, authoring environments, enterprise information portals, search engines, experts directories, collaborative tools, and in-process decision capture. These systems, together, comprise JPL's architecture to capture, organize, store, and distribute key learnings for the U.S. exploration of space.

knowledge management organizational learning NASA ↗

NASA NTRS2006

Near-Miss Evaluation Bias as an Obstacle to Organizational Learning: Lessons from NASA

After the Shuttle Columbia catastrophe, the investigation board (CAIB) stated that NASA needs to develop a "learning culture", meaning a capability to learn from past failures by understanding the technical and organizational causes of these mistakes (CAIB report, 2003). While many organizations learn from obvious failures, we argue that it is harder for organizations to learn from near-miss events (i.e., situations where a failure does not occur but nearly did), because these near-misses are processed as successes. For the shuttle program, prior debris problems could have caused a similar failure as on the Columbia mission except that the large pieces missed the highly sensitive portions of the orbiter. This acceptance of foam debris was adopted as a normal occurrence by the shuttle program managers similar to the problems at the time of the Challenger Disaster (detailed in Vaughan, 1996). We extend that work to show that an outcome bias influences people's evaluation of project managers, such that managers of failed missions were perceived more poorly than managers who made the same decisions but whose mission ended in either success or a near-miss. The similarity of ratings between the near-miss and success condition imply that even when a problem occurs that is clearly linked to prior managerial decisions, if the project is not harmed because of good luck, that manager is not held accountable for faculty decision making and neither the individual manager nor the organization learn from the experience potentially increasing the likelihood of a failure in the future.

Dillon, Robin L.↗

NASA NTRS2024

NASA Risk Management Handbook: Version 2.0, Part 1

The purpose of this handbook is to provide an in-depth reference for the practice of risk management in NASA, updating the guidance offered in its original version, NASA/SP-2011-3422 (November 2011), and closely aligning the updated guidance with the current NASA Procedural Requirements for Agency Risk Management, NPR 8000.4, and the parent NASA Policy Directive for NASA Governance and Strategic Management, NPD 1000.0. NPD 1000.0 introduces with emphasis the concept of “Risk Leadership,” making it a fundamental tenet and pillar of the risk management culture that it advocates for the Agency. NPR 8000.4 applies this concept and establishes Risk Management (RM) requirements for the Agency as an integrated enterprise, as well as the RM requirements for portfolio elements within the enterprise. Such elements include the various programs and projects that contribute to the Agency’s objectives and the various institutional activities carried out by entities that contribute to mission support. The present version of the handbook also emphasizes the integration of risk management processes across activity and project life cycles and their coordination and interaction with day-to-day programmatic and organizational functions. Areas of application of risk assessment and management that were not covered with specific guidance in the preceding version are addressed in this version with in-depth examples. The handbook is structured into two parts, whose chapters are in turn organized in a sequential order intended to facilitate a gradual and progressive introduction of the reader to risk management principles and practices. Part 1 of the handbook is dedicated to the introduction of the basic foundations of the NASA integrated risk management framework, the related fundamental risk concepts, the description of the risk management and decision processes that are to be implemented within the framework, the discussion of the risk assessment techniques that should be utilized in support of such processes, and the management and organizational interactions and interfaces that should be enabled to implement an effective integration of risk management activities within the Agency. Part 2 provides self-contained, end-to-end examples of application of the processes and techniques introduced in Part 1, in the context of both programmatic (i.e., project and/or mission related) and institutional activities.

Uncertainty↗

NASA NTRS2024

NASA Risk Management Handbook: Version 2.0, Part 2

The purpose of this handbook is to provide an in-depth reference for the practice of risk management in NASA, updating the guidance offered in its original version, NASA/SP-2011-3422 (November 2011), and closely aligning the updated guidance with the current NASA Procedural Requirements for Agency Risk Management, NPR 8000.4, and the parent NASA Policy Directive for NASA Governance and Strategic Management, NPD 1000.0 (January 2020). NPD 1000.0 introduces with emphasis the concept of “Risk Leadership,” making it a fundamental tenet and pillar of the risk management culture that it advocates for the Agency. NPR 8000.4 applies this concept and establishes Risk Management (RM) requirements for the Agency as an integrated enterprise, as well as the RM requirements for portfolio elements within the enterprise. Such elements include the various programs and projects that contribute to the Agency’s objectives and the various institutional activities carried out by entities that contribute to mission support. The present version of the handbook also emphasizes the integration of risk management processes across activity and project life cycles and their coordination and interaction with day-to-day programmatic and organizational functions. Areas of application of risk assessment and management that were not covered with specific guidance in the preceding version are addressed in this version with in-depth examples. The handbook is structured into two parts, whose chapters are in turn organized in a sequential order intended to facilitate a gradual and progressive introduction of the reader to risk management principles and practices. Part 1 of the handbook is dedicated to the introduction of the basic foundations of the NASA integrated risk management framework, the related fundamental risk concepts, the description of the risk management and decision processes that are to be implemented within the framework, the discussion of the risk assessment techniques that should be utilized in support of such processes, and the management and organizational interactions and interfaces that should be enabled to implement an effective integration of risk management activities within the Agency. Part 2 provides self-contained, end-to-end examples of application of the processes and techniques introduced in Part 1, in the context of both programmatic (i.e., project and/or mission related) and institutional activities.

Risk Leadership↗

NASA NTRS2010

Adaptive Planning: Understanding Organizational Workload to Capability/ Capacity through Modeling and Simulation

In August 2003, the Secretary of Defense (SECDEF) established the Adaptive Planning (AP) initiative [1] with an objective of reducing the time necessary to develop and revise Combatant Commander (COCOM) contingency plans and increase SECDEF plan visibility. In addition to reducing the traditional plan development timeline from twenty-four months to less than twelve months (with a goal of six months)[2], AP increased plan visibility to Department of Defense (DoD) leadership through In-Progress Reviews (IPRs). The IPR process, as well as the increased number of campaign and contingency plans COCOMs had to develop, increased the workload while the number of planners remained fixed. Several efforts from collaborative planning tools to streamlined processes were initiated to compensate for the increased workload enabling COCOMS to better meet shorter planning timelines. This paper examines the Joint Strategic Capabilities Plan (JSCP) directed contingency planning and staffing requirements assigned to a combatant commander staff through the lens of modeling and simulation. The dynamics of developing a COCOM plan are captured with an ExtendSim [3] simulation. The resulting analysis provides a quantifiable means by which to measure a combatant commander staffs workload associated with development and staffing JSCP [4] directed contingency plans with COCOM capability/capacity. Modeling and simulation bring significant opportunities in measuring the sensitivity of key variables in the assessment of workload to capability/capacity analysis. Gaining an understanding of the relationship between plan complexity, number of plans, planning processes, and number of planners with time required for plan development provides valuable information to DoD leadership. Through modeling and simulation AP leadership can gain greater insight in making key decisions on knowing where to best allocate scarce resources in an effort to meet DoD planning objectives.

Hase, Chris↗

NASA NTRSDate not supplied

Habitability Assessments And Lessons-learned From 3-day And 11-day Enriched Oxygen Hypobaric Chamber Tests At NASA Johnson Space Center

INTRODUCTION: Decompression sickness (DCS) is a risk to the health and performance of astronauts and high-altitude aircrew. Tolerance to flammability, hypoxia, prebreathe duration, and DCS risk varies across different organizations, vehicles, suits, and destinations, necessitating a variety of DCS risk mitigation approaches. Existing models of altitude DCS risk are often insufficient to enable accurate risk-informed decisions during hardware development, mission planning, and flight operations. METHODS: NASA completed outfitting of a dedicated facility at Johnson Space Center to support testing of up to eight human subjects for multiple days in hypobaric and enriched oxygen atmospheres. The primary purpose of the testing capability is validation of DCS risk mitigation protocols for Artemis missions to the Moon; however, it will also support development and validation of a generalizable altitude DCS risk estimation tool. A 3-day and an 11-day prebreathe validation test were completed in 2022, each with 8 human subjects living at 56.5 kPa (8.2 psia), 34% O2, 66% N2, with 5 simulated EVAs performed on masks at 29.6 kPa (4.3 psi), 85% O2, 15% N2. Facility and organizational lessons-learned and process improvements were recorded during and following the tests, and subjective habitability ratings were recorded daily during the 11-day test. Hypoxia and DCS-related physiological and cognitive outcome measures were recorded during both tests and are reported in companion presentations. RESULTS & DISCUSSION: All subjects completed each of the tests. Primary habitability issues related to mask discomfort during simulated EVAs and poor sleep quality due to thin mattresses. Polybenzimidazole (PBI) clothing was worn by all subjects due to the increased fire risk and may be required for Artemis missions; clothing was found to be acceptable overall with the worst ratings being due to poor fit and inelasticity. Chamber O2 and CO2 sensor inconsistency was observed that did not result in test termination but required post-test follow-up. Forward plans include additional hypobaric testing and integration of existing and future physiological outcome data into an open-source Aerospace Estimation Tool for Hypobaric Exposure Risk (AETHER). NASA is also working to make the testing capability available to commercial companies.

Andrew F J Abercromby↗

NASA NTRSDate not supplied

Habitability Assessments And Lessons-learned From 3-day And 11-day Enriched Oxygen Hypobaric Chamber Tests At NASA Johnson Space Center

INTRODUCTION: Decompression sickness (DCS) is a risk to the health and performance of astronauts and high-altitude aircrew. Tolerance to flammability, hypoxia, prebreathe duration, and DCS risk varies across different organizations, vehicles, suits, and destinations, necessitating a variety of DCS risk mitigation approaches. Existing models of altitude DCS risk are often insufficient to enable accurate risk-informed decisions during hardware development, mission planning, and flight operations. METHODS: NASA completed outfitting of a dedicated facility at Johnson Space Center to support testing of up to eight human subjects for multiple days in hypobaric and enriched oxygen atmospheres. The primary purpose of the testing capability is validation of DCS risk mitigation protocols for Artemis missions to the Moon; however, it will also support development and validation of a generalizable altitude DCS risk estimation tool. A 3-day and an 11-day prebreathe validation test were completed in 2022, each with 8 human subjects living at 56.5 kPa (8.2 psia), 34% O2, 66% N2, with 5 simulated EVAs performed on masks at 29.6 kPa (4.3 psi), 85% O2, 15% N2. Facility and organizational lessons-learned and process improvements were recorded during and following the tests, and subjective habitability ratings were recorded daily during the 11-day test. Hypoxia and DCS-related physiological and cognitive outcome measures were recorded during both tests and are reported in companion presentations. RESULTS & DISCUSSION: All subjects completed each of the tests. Primary habitability issues related to mask discomfort during simulated EVAs and poor sleep quality due to thin mattresses. Polybenzimidazole (PBI) clothing was worn by all subjects due to the increased fire risk and may be required for Artemis missions; clothing was found to be acceptable overall with the worst ratings being due to poor fit and inelasticity. Chamber O2 and CO2 sensor inconsistency was observed that did not result in test termination but required post-test follow-up. Forward plans include additional hypobaric testing and integration of existing and future physiological outcome data into an open-source Aerospace Estimation Tool for Hypobaric Exposure Risk (AETHER). NASA is also working to make the testing capability available to commercial companies.

Andrew Abercromby↗

NASA NTRSDate not supplied

Standards Roadmapping for Mission Assurance in Commercial Spaceflight

This paper explores the application of established roadmapping approaches in standards planning and roadmap development to support mission assurance objectives. Mission assurance requires proactive planning across multiple domains including technical, schedule, cost, organizational, and policy. The intersection of these domains happens at the enterprise level, and enterprise systems engineering processes can systematically guide mission assurance activities, including development of standards roadmaps. Standards planning and development is a critical contributor to an effective mission assurance strategy, although standards are often developed and adopted reactively. Technology and standards planning is a key Enterprise Process Management activity, as identified by the Systems Engineering Body of Knowledge, and technology and standards roadmaps are strategic tools used to enable that process. Technology roadmapping has proven effective in guiding enterprise architecture and concept design, and standards planning supports enterprise requirements definition and management. However, there are limited methodologies to guide such standards planning activities. The application of technology roadmapping principles to standards planning formalizes the process and establishes a repeatable framework for standards development and integration. The framework development described in this paper fills a methodological gap by adapting proven roadmapping techniques to standards planning. Through a comparison of roadmapping methodologies and a case study analysis, this approach outlines a systematic process for anticipating and planning standards needs. In the case of developing a standards integration roadmap for NASA’s Office of Safety and Mission Assurance (OSMA), technology roadmapping principles were adapted to develop a roadmap framework to allow NASA and OSMA to define an agency-wide standards integration plan for commercial and industry safety and mission assurance standards. The outcome highlights key adaptations required for roadmapping in a standards context and describes the process steps to formalize a standards planning activity. The roadmapping framework adapted for standards development and integration enables more strategic, forward-looking standards planning and allows for better alignment between evolving capabilities and mission assurance requirements. Beyond its application to mission assurance, this methodology can be transferred to other domains requiring standards development, providing a systematic approach that integrates systems engineering principles into policy strategy and decision-making processes.

Enterprise Systems Engineering↗

NASA NTRSDate not supplied

Developing Concepts of Operations Using Multi-Step Tool Techniques With Large Language Models

The National Aeronautics and Space Administration (NASA) Air Mobility Pathfinders (AMP) project is developing and evaluating concepts of operations (ConOps) for safe, secure, and scalable Urban Air Mobility (UAM) operations. The AMP project’s Operational Concepts, Architecture, and Requirements Integration (OCARI) Team is using a Model Based System Engineering (MBSE) approach for integration, interoperability, and traceability of Advanced Air Mobility (AAM) ecosystems centered around urban air taxi services. The team’s goal is to define structures and behaviors needed for system feasibility, readiness, and interoperability, establish a UAM knowledge base, and trace and validate assumptions and requirements relevant to AAM. NASA Langley Research Center (LaRC) is spearheading an innovative digital engineering approach to integrate, communicate, and facilitate the research of multi-modal transportation systems. The Knowledge-based Digital Platform (KbDP) is a concept being developed that ties the workflows of Project Managers (PM), Principal Investigators (PI), and System Engineers together across organizational boundaries. It does so through the management of an information database defined by mathematical, data science, and system engineering principles. Machine Learning (ML) algorithms play a key role in this concept by extracting meaningful knowledge from relational and graph databases, document repositories, and system artifacts, which the human user leverages to greatly improve the efficiency and effectiveness of their research. Recent advancements in the field of Large Language Models (LLMs), specifically models trained for tool use, such as Command-R , now allow for the reliable implementation of single-step and multi-step tool-centric systems. These techniques provide the LLM with a set of tools, in our case Python functions, that can be called on to answer a much wider range of questions compared to LLMs implemented using a traditional single-source or Retrieval Augmented Generation (RAG) approach. Through this method, the LLM can pull information from multiple data sources, such as relational or graph databases, document repositories, application programming interfaces (APIs), and SysML artifacts depending on the user’s question. The LLM can also output the information in a variety of different formats, using output generation tools, such as CSV, UML, or SysML artifacts. Additionally, tools can be assigned roles and can work together to provide answers to queries in an “agent” like approach, similar to that implemented by Microsoft’s AutoGen framework where different agents can converse with each other to accomplish tasks. Previously, our team developed a chatbot system with “agent like” functionality in the form of different “modes” the user could select from a user interface (UI), this architecture can be seen on the left in figure 1. Three different modes were implemented, the first mode allowed the LLM to utilize the structures and algorithms within a graph database to trace UAM requirements. The second mode gave the LLM access to a vector search capable of providing relevant information from thousands of document pages related to UAM ConOps and requirements. The third mode served as a general assistant where users could enter open-ended questions and custom prompts to utilize the LLM for different use-cases. This system improved the process surrounding generating and analyzing information related to UAM requirements, however, the implementation provided a clunky user experience. Users were required to know what mode to select within the UI in advance before entering their question to the selected tool. Moreover, the different tools were isolated from each other, they lacked bidirectional links that would allow for tools to collaborate to generate better responses. Our team is working on a new architecture, seen on the right in the below figure, with the goal to address many of the UX shortcomings of our original system while improving the accuracy and depth of responses from the LLM. This new system will automatically select the appropriate tool to use based off the user’s question. Each tool will be capable of calling on any of the other tools available to the LLM, resulting in a collaborative pipeline where tools can pass data between other tools until enough data is received to generate an answer to the user’s question. Using a locally deployed, open-source, LLM, the NASA OCARI team, in collaboration with Collins Aerospace, will implement a prototype application that will bridge knowledge across multiple sources to assist System Engineers (SEs) with requirements discovery and tracing, research question and use case identification, and assumption validation. Such a system will also allow SEs to more easily, and intuitively, explore the AAM ecosystem, ultimately improving the efficiency and effectiveness of the SE's research and decision-making processes surrounding ConOps development and validation. In this session, our team will provide a video demonstration of our new prototype architecture in action. We will also present an overview of our prototype system architecture and talk about its advantages over traditional LLM deployments along with how those advantages can provide additional value to the field of System Engineering.

systems engineering↗