SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
This viewgraph presentation describes a case study as a model for integrated project management. The ISS Program Office (ISSPO) developed replacement fluid filtration cartridges in house for the International Space Station (ISS). The presentation includes a step-by-step procedure and organizational charts for how the fluid filtration problem was approached.
Return of samples from the Moon via the Apollo 11, 12 and 14 missions represented NASA's first attempts into conducting what is now known as Restricted Earth Return. When Apollo returned samples from the moon, there were few of the agencies, legal and organizational structures that exist now. A half a century later, the U.S system has seen changes in legal and organizational structures with the appearance of different agencies and legal requirements. During the Apollo era, U.S. legal and organizational structures, assessed threats to the Earth’s biosphere with a priority towards the risks of effects from extraterrestrial pathogens on humans, though the guiding document, the Baylor Protocol, also considered animal and plant models. This presentation focuses on a review of two main questions: Did the return of samples from the Moon happen at a “sweet spot” in time, where the number of requirements and conservatism was commensurate with the scientific knowns and unknowns? Could the emphasis of risk mitigation on human pathogens during the return of extraterrestrial samples be more about human solipsism, leaving us with a blind spot to sources of pathogens, such as fungi, which could lead to an impact to our biosphere, and we humans, via our food chains and agricultural pathways?
In the present technological society, there is a major need to build machines that would execute intelligent tasks operating in uncertain environments with minimum interaction with a human operator. Although some designers have built smart robots, utilizing heuristic ideas, there is no systematic approach to design such machines in an engineering manner. Recently, cross-disciplinary research from the fields of computers, systems AI and information theory has served to set the foundations of the emerging area of the design of intelligent machines. Since 1977 Saridis has been developing an approach, defined as Hierarchical Intelligent Control, designed to organize, coordinate and execute anthropomorphic tasks by a machine with minimum interaction with a human operator. This approach utilizes analytical (probabilistic) models to describe and control the various functions of the intelligent machine structured by the intuitively defined principle of Increasing Precision with Decreasing Intelligence (IPDI) (Saridis 1979). This principle, even though resembles the managerial structure of organizational systems (Levis 1988), has been derived on an analytic basis by Saridis (1988). The purpose is to derive analytically a Boltzmann machine suitable for optimal connection of nodes in a neural net (Fahlman, Hinton, Sejnowski, 1985). Then this machine will serve to search for the optimal design of the organization level of an intelligent machine. In order to accomplish this, some mathematical theory of the intelligent machines will be first outlined. Then some definitions of the variables associated with the principle, like machine intelligence, machine knowledge, and precision will be made (Saridis, Valavanis 1988). Then a procedure to establish the Boltzmann machine on an analytic basis will be presented and illustrated by an example in designing the organization level of an Intelligent Machine. A new search technique, the Modified Genetic Algorithm, is presented and proved to converge to the minimum of a cost function. Finally, simulations will show the effectiveness of a variety of search techniques for the intelligent machine.
For several decades, Multidisciplinary Design Optimization (MDO) has served an important role in aerospace engineering by incorporating physics based disciplinary models into integrated system or sub-system models for use in research, development, (R&D) and design. This paper examines MDO's role in facilitating the integration of the researchers from different single disciplines during R&D and early design of large-scale complex engineered systems (LaCES) such as aerospace systems. The findings in this paper are summarized from a larger study on interdisciplinary practices and perspectives that included considerable empirical data from surveys, interviews, and ethnography. The synthesized findings were derived by integrating the data with theories from organization science and engineering. The over-arching finding is that issues related to cognition, organization, and social interrelations mostly dominate interactions across disciplines. Engineering issues, such as the integration of hardware or physics-based models, are not as significant. Correspondingly, the data showed that MDO is not the primary integrator of researchers working across disciplines during R&D and early design of LaCES. Cognitive focus such as analysis versus design, organizational challenges such as incentives, and social opportunities such as personal networks often drove the human interactive practices among researchers from different disciplines. Facilitation of the inherent confusion, argument, and learning in crossdisciplinary research was identified as one of several needed elements of enabling successful research across disciplines.
Over the last 25 years, a tremendous progress has been made in the Earth science space-based remote sensing observations, technologies and algorithms. Such advancements have improved the predictability by providing lead-time and accuracy of forecast in weather, climate, natural hazards, and natural resources. It has further reduced or bounded the overall uncertainties by partially improving our understanding of planet Earth as an integrated system that is governed by non-linear and chaotic behavior. Many countries such US, European Community, Japan, China and others have invested billions of dollars in developing and launching space-based assets in the low earth (LEO) and geostationary (GEO) orbits. However, the wealth of this scientific knowledge that has potential of extracting monumental socio-economic benefits from such large investments have been slow in reaching to public and decision makers. For instance, there are a number of areas such as energy forecasting, aviation safety, agricultural competitiveness, disaster management, security, air quality and public health can directly take advantage. Nevertheless, we all live in a global economy that depends on access to the best available Earth Science information for all inhabitants of this planet. This paper surveys and examines a number such applications in terms of their architecture, maturity and economic applicability as they apply to the societal needs. A detailed analysis is also presented of various challenges and issues that pertain to a number of areas such as: (1) difficulties in making a speedy transition of data and information from observations and models to relevant Decision Support Systems (DSS) or tools, (2) data and models inter-operability issues, (3) limitations of spatial, spectral and temporal resolution, (4) communication limitations as dictated by the availability of image processing and data compression techniques. Additionally, the most critical element amongst all is the organizational and management boundaries that must be resolved at local, state, national and international levels to implement and realize free flow of such vital information. This paper also makes attempts to address this topic and discuss possible approaches to deal with this quandary.
Over the last four decades a tremendous progress has been made in the Earth science space-based remote sensing observations, technologies and algorithms. Such advancements have improved the predictability by providing lead-time and accuracy of forecast in weather, climate, natural hazards, and natural resources. It has further reduced or bounded the overall uncertainties by partially improving our understanding of planet Earth as an integrated system that is governed by non-linear and chaotic behavior. Many countries such as the US, European Community, Japan, China, Russia, India has and others have invested billions of dollars in developing and launching space-based assets in the low earth (LEO) and geostationary (GEO) orbits. However, the wealth of this scientific knowledge that has potential of extracting monumental socio-economic benefits from such large investments have been slow in reaching the public and decision makers. For instance, there are a number of areas such as water resources and availability, energy forecasting, aviation safety, agricultural competitiveness, disaster management, air quality and public health, which can directly take advantage. Nevertheless, we all live in a global economy that depends on access to the best available Earth Science information for all inhabitants of this planet. This presentation discusses a process to transition Earth science data and products for societal needs including NASA's experience in achieving such objectives. It is important to mention that there are many challenges and issues that pertain to a number of areas such as: (1) difficulties in making a speedy transition of data and information from observations and models to relevant Decision Support Systems (DSS) or tools, (2) data and models inter-operability issues, (3) limitations of spatial, spectral and temporal resolution, (4) communication limitations as dictated by the availability of image processing and data compression techniques. Additionally, the most critical element amongst all is the organizational and management boundaries that must be resolved at local, state, national and international levels to implement and realize free flow of such vital information.
The combination of increased electric load growth across all sectors, deferred electrical infrastructure investment, and other factors resulting in variable electric power supply, has created technical challenges to maintaining a resilient and reliable grid. Many federal, regional, and local efforts are in play to modernize the electric grid, including advancing building technologies and distributed energy resources (DERs) that are utilizing smarter controls to become responsive to both occupant and grid needs. This report reviews ten pilot projects demonstrating how groups of buildings combined with behind-the-meter (BTM) DERs such as electric vehicle (EV) charging, battery storage, flexible HVAC and domestic hot water systems, and photovoltaic systems can reliably and cost effectively provide grid services. Each of the ten pilot projects aim to deliver both energy efficiency and demand flexibility (DF) while supporting load growth. The ten demonstration teams are piloting flexible DER packages across diverse communities of residential and commercial buildings to address a variety of regional grid needs. The outcomes of these pilot projects will be used to inform future scaling through utility program development. This paper characterizes the ten teams, showcasing the decision-making process used by each group to develop their packages (Section 2), the grid services they plan to deliver (Section 3), the types of DER packages selected for deployment within building sectors (Section 4) and trends between building sector, DER types, and grid services In order to achieve community scale benefits, the pilot projects must utilize aggregated control mechanisms for coordinating buildings and DERs together. Several types of coordinated control architectures have evolved amongst the teams, influenced by use type, existing market conditions, and integration type. Three coordinated controls architectures have been characterized, highlighting their use cases, benefits, challenges, and tradeoffs in their design. These insights can aid utilities, control vendors, and developers in scaling community-level energy systems (Paul, 2024). Ultimately, the technology packages selected by the ten teams will be coordinated to provide power system services, also known as grid services. Insights from these demonstrations will be useful for grid operators, regulators, aggregators and other stakeholders as they look to deploy demand flexible resources as grid services in the future. The grid services that each team is targeting for demonstration are described in Section 3 and Section 4. Methods for evaluating the grid services have been described in the paper Metrics for Evaluating Grid Service Provision from Communities of Grid-interactive and Efficient Buildings and other DER (MacDonald, 2023). To identify technology packages for demonstration, Section 2 shows that project teams used a range of analysis approaches, including building energy modeling, AMI data analysis, cost-benefit frameworks, and utility pilot data. Some teams emphasized technical modeling to quantify grid impacts and demand reduction potential, while others prioritized economic evaluations, stakeholder input, or exploratory pilots to inform deployment decisions. This diversity reflects the need to tailor selection methods to project goals, available data, and organizational context. Section 5 discusses trends between the DER technologies deployed and the grid service provisions from each team. Residential buildings (multifamily and single family) lean towards technologies that enhance energy efficiency (e.g. weatherization upgrades, smart thermostats) and onsite power generation integration (e.g. solar PV). Commercial building demonstrations prioritize technologies that ensure operational reliability (e.g. battery storage) and centralized energy management systems and optimization solutions. Teams that are deploying controllable storage-based technologies are more likely to provide grid services that require a near real-time response. Teams incorporating load shifting technologies like smart thermostats with HEMs are likely to include energy markets participation and customer bill management offerings. Campus demonstrations are adopting diverse sets of DERs to emphasize renewable generation, paired with centralized control. This section also describes technologies that were considered during project planning but ultimately excluded from final deployment. These demonstrations reveal that effective DER package design should be tailored to building type, customer segment, and construction vintage. Multifamily buildings benefit from centralized HVAC upgrades and supervisory controls, while single-family homes are well-suited for individualized technologies like solar, storage, and smart home energy monitors. Commercial and campus settings prioritize EMIS integration and load optimization. New construction enables cost-effective integration of DER-ready infrastructure, whereas retrofits require deployments aligned with owner and tenant value streams. For utility program planners, early coordination with developers and building owners, paired with segmented and modular program offerings, can improve adoption, scalability, and grid impact.
As part of its award-winning software process improvement program, the Software Engineering Laboratory (SEL) has developed an effective method for packaging organizational best practices based on real project experience into useful handbooks and training courses. This paper shares the SEL's experience over the past 12 years creating and updating software process handbooks and training courses. It provides cost models and guidelines for successful experience packaging derived from SEL experience.
The increasing complexity of engineering systems has sparked increasing interest in multidisciplinary optimization (MDO). This paper presents a survey of recent publications in the field of aerospace where interest in MDO has been particularly intense. The two main challenges of MDO are computational expense and organizational complexity. Accordingly the survey is focussed on various ways different researchers use to deal with these challenges. The survey is organized by a breakdown of MDO into its conceptual components. Accordingly, the survey includes sections on Mathematical Modeling, Design-oriented Analysis, Approximation Concepts, Optimization Procedures, System Sensitivity, and Human Interface. With the authors' main expertise being in the structures area, the bulk of the references focus on the interaction of the structures discipline with other disciplines. In particular, two sections at the end focus on two such interactions that have recently been pursued with a particular vigor: Simultaneous Optimization of Structures and Aerodynamics, and Simultaneous Optimization of Structures Combined With Active Control.
The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.
The Joint Polar Satellite System is a modern, large-scale, complex, multi-mission aerospace program, and presents a variety of design, testing and operational challenges due to: (1) System Scope: multi-mission coordination, role, responsibility and accountability challenges stemming from porous/ill-defined system and organizational boundaries (including foreign policy interactions) (2) Degree of Concurrency: design, implementation, integration, verification and operation occurring simultaneously, at multiple scales in the system hierarchy (3) Multi-Decadal Lifecycle: technical obsolesce, reliability and sustainment concerns, including those related to organizational and industrial base. Additionally, these systems tend to become embedded in the broader societal infrastructure, resulting in new system stakeholders with perhaps different preferences (4) Barriers to Effective Communications: process and cultural issues that emerge due to geographic dispersion and as one spans boundaries including gov./contractor, NASA/Other USG, and international relationships.
Create an improved method to calculate reliability of a conceptual launch vehicle system prior to fabrication by using historic data of actual root causes of failures. While failures have unique "proximate causes", there are typically a finite amount of common "root causes". Heretofore launch vehicle reliability evaluation typically hardware-centric statistical analyses, while most root causes of failures are been shown to be human-centric. A method based on human-centric root causes can be used to quantify reliability assessments and focus proposed actions to mitigate problems. Existing methods have been optimistic in their projections of launch vehicle reliability compared to actuals. Hypothesis: reliability of a conceptual launch vehicle can be more accurately evaluated based on a rational, probabilistic approach using past failure assessment teams' findings predicated on human-centric causes."Human Reliability Analysis Methods Selection Guidance for NASA"Chandler F.T., et al., NASA HQ/OSMA study group, July 2006. Outside HRA experts from academia, other federal labs, and the private sector. 50 system reliability methods considered, fourteen selected for further study, four finally selected as best suited for human spaceflight. Probabilistic Risk Analysis (PRA) + Human Reliability Analysis (HRA) enabled incorporating effects and probabilities of human errors. While four down-selected methods deemed appropriate for failure assessment, it did not appear that these methods could be concisely applied to perform major system-wide assessment of probability of failure of a conceptual design without becoming unwieldy."Engineering a Safer World", Detailed, comprehensive study external to NASA Leveson N. G., MIT, 2011.Systems-Theoretic Accident Model and Processes (STAMP). All-encompassing accident model based on systems theory analyzed accidents after they occurred and created approaches to prevent occurrence in developing systems not focused on failure prevention per se, but rather reducing hazards by influencing human behavior through use of constraints, hierarchical control structures, and process models to improve system safetySystem Theoretic Process Analysis (STPA) addresses predictive part of problem (a "hazard analysis"). Includes all causal factors identified in STAMP: "...design errors, software flaws, component interaction accidents, cognitively complex human decision-making errors, and social organizational and management factors contributing to accidents" can guide design process rather than require it to exist before-hand did not appear capable of concise application for system-wide assessment of probability of failure of a conceptual design without becoming unwieldy.
This paper presents the results of a 2012 life cycle cost (LCC) study of hybrid Reusable Booster Systems (RBS) conducted by NASA Kennedy Space Center (KSC) and the Air Force Research Laboratory (AFRL). The work included the creation of a new cost estimating model and an LCC analysis, building on past work where applicable, but emphasizing the integration of new approaches in life cycle cost estimation. Specifically, the inclusion of industry processes/practices and indirect costs were a new and significant part of the analysis. The focus of LCC estimation has traditionally been from the perspective of technology, design characteristics, and related factors such as reliability. Technology has informed the cost related support to decision makers interested in risk and budget insight. This traditional emphasis on technology occurs even though it is well established that complex aerospace systems costs are mostly about indirect costs, with likely only partial influence in these indirect costs being due to the more visible technology products. Organizational considerations, processes/practices, and indirect costs are traditionally derived ("wrapped") only by relationship to tangible product characteristics. This traditional approach works well as long as it is understood that no significant changes, and by relation no significant improvements, are being pursued in the area of either the government acquisition or industry?s indirect costs. In this sense then, most launch systems cost models ignore most costs. The alternative was implemented in this LCC study, whereby the approach considered technology and process/practices in balance, with as much detail for one as the other. This RBS LCC study has avoided point-designs, for now, instead emphasizing exploring the trade-space of potential technology advances joined with potential process/practice advances. Given the range of decisions, and all their combinations, it was necessary to create a model of the original model and use genetic algorithms to explore results. A strong business case occurs when viable paths are identified for an affordable up-front investment, and these paths can credibly achieve affordable, responsive operations, characterized by smaller direct touch labor efforts at the wing level from flight to flight. The results supporting this approach, its potential, and its conclusions are presented here.
This paper presents the results of a 2012 life cycle cost (LCC) study of hybrid Reusable Booster Systems (RBS) conducted by NASA Kennedy Space Center (KSC) and the Air Force Research Laboratory (AFRL). The work included the creation of a new cost estimating model and an LCC analysis, building on past work where applicable, but emphasizing the integration of new approaches in life cycle cost estimation. Specifically, the inclusion of industry processes/practices and indirect costs were a new and significant part of the analysis. The focus of LCC estimation has traditionally been from the perspective of technology, design characteristics, and related factors such as reliability. Technology has informed the cost related support to decision makers interested in risk and budget insight. This traditional emphasis on technology occurs even though it is well established that complex aerospace systems costs are mostly about indirect costs, with likely only partial influence in these indirect costs being due to the more visible technology products. Organizational considerations, processes/practices, and indirect costs are traditionally derived ("wrapped") only by relationship to tangible product characteristics. This traditional approach works well as long as it is understood that no significant changes, and by relation no significant improvements, are being pursued in the area of either the government acquisition or industry?s indirect costs. In this sense then, most launch systems cost models ignore most costs. The alternative was implemented in this LCC study, whereby the approach considered technology and process/practices in balance, with as much detail for one as the other. This RBS LCC study has avoided point-designs, for now, instead emphasizing exploring the trade-space of potential technology advances joined with potential process/practice advances. Given the range of decisions, and all their combinations, it was necessary to create a model of the original model and use genetic algorithms to explore results. A strong business case occurs when viable paths are identified for an affordable up-front investment, and these paths can credibly achieve affordable, responsive operations, characterized by smaller direct touch labor efforts at the wing level from flight to flight. The results supporting this approach, its potential, and its conclusions are presented here.
Introduction: The Integrated Medical Model (IMM) Project represents one aspect of NASA's Human Research Program (HRP) to quantitatively assess medical risks to astronauts for existing operational missions as well as missions associated with future exploration and commercial space flight ventures. The IMM takes a probabilistic approach to assessing the likelihood and specific outcomes of one hundred medical conditions within the envelope of accepted space flight standards of care over a selectable range of mission capabilities. A specially developed Integrated Medical Evidence Database (iMED) maintains evidence-based, organizational knowledge across a variety of data sources. Since becoming operational in 2011, version 3.0 of the IMM, the supporting iMED, and the expertise of the IMM project team have contributed to a wide range of decision and informational processes for the space medical and human research community. This presentation provides an overview of the IMM conceptual architecture and range of application through examples of actual space flight community questions posed to the IMM project. Methods: Figure 1 [see document] illustrates the IMM modeling system and scenario process. As illustrated, the IMM computational architecture is based on Probabilistic Risk Assessment techniques. Nineteen assumptions and limitations define the IMM application domain. Scenario definitions include crew medical attributes and mission specific details. The IMM forecasts probabilities of loss of crew life (LOCL), evacuation (EVAC), quality time lost during the mission, number of medical resources utilized and the number and type of medical events by combining scenario information with in-flight, analog, and terrestrial medical information stored in the iMED. In addition, the metrics provide the integrated information necessary to estimate optimized in-flight medical kit contents under constraints of mass and volume or acceptable level of mission risk. Results and Conclusions: Historically, IMM simulations support Science and Technology planning, Exploration mission planning, and ISS program operations by supplying simulation support, iMED data information, and subject matter expertise to Crew Health and Safety and the HRP. Upcoming release of IMM version 4.0 seeks to provide enhanced functionality to increase the quality of risk decisions made using the IMM through a more accurate representation of the real world system.
The National Aeronautics and Space Administration (NASA) Air Mobility Pathfinders (AMP) project is developing and evaluating concepts of operations (ConOps) for safe, secure, and scalable Urban Air Mobility (UAM) operations. The AMP project’s Operational Concepts, Architecture, and Requirements Integration (OCARI) Team is using a Model Based System Engineering (MBSE) approach for integration, interoperability, and traceability of Advanced Air Mobility (AAM) ecosystems centered around urban air taxi services. The team’s goal is to define structures and behaviors needed for system feasibility, readiness, and interoperability, establish a UAM knowledge base, and trace and validate assumptions and requirements relevant to AAM. NASA Langley Research Center (LaRC) is spearheading an innovative digital engineering approach to integrate, communicate, and facilitate the research of multi-modal transportation systems. The Knowledge-based Digital Platform (KbDP) is a concept being developed that ties the workflows of Project Managers (PM), Principal Investigators (PI), and System Engineers together across organizational boundaries. It does so through the management of an information database defined by mathematical, data science, and system engineering principles. Machine Learning (ML) algorithms play a key role in this concept by extracting meaningful knowledge from relational and graph databases, document repositories, and system artifacts, which the human user leverages to greatly improve the efficiency and effectiveness of their research. Recent advancements in the field of Large Language Models (LLMs), specifically models trained for tool use, such as Command-R , now allow for the reliable implementation of single-step and multi-step tool-centric systems. These techniques provide the LLM with a set of tools, in our case Python functions, that can be called on to answer a much wider range of questions compared to LLMs implemented using a traditional single-source or Retrieval Augmented Generation (RAG) approach. Through this method, the LLM can pull information from multiple data sources, such as relational or graph databases, document repositories, application programming interfaces (APIs), and SysML artifacts depending on the user’s question. The LLM can also output the information in a variety of different formats, using output generation tools, such as CSV, UML, or SysML artifacts. Additionally, tools can be assigned roles and can work together to provide answers to queries in an “agent” like approach, similar to that implemented by Microsoft’s AutoGen framework where different agents can converse with each other to accomplish tasks. Previously, our team developed a chatbot system with “agent like” functionality in the form of different “modes” the user could select from a user interface (UI), this architecture can be seen on the left in figure 1. Three different modes were implemented, the first mode allowed the LLM to utilize the structures and algorithms within a graph database to trace UAM requirements. The second mode gave the LLM access to a vector search capable of providing relevant information from thousands of document pages related to UAM ConOps and requirements. The third mode served as a general assistant where users could enter open-ended questions and custom prompts to utilize the LLM for different use-cases. This system improved the process surrounding generating and analyzing information related to UAM requirements, however, the implementation provided a clunky user experience. Users were required to know what mode to select within the UI in advance before entering their question to the selected tool. Moreover, the different tools were isolated from each other, they lacked bidirectional links that would allow for tools to collaborate to generate better responses. Our team is working on a new architecture, seen on the right in the below figure, with the goal to address many of the UX shortcomings of our original system while improving the accuracy and depth of responses from the LLM. This new system will automatically select the appropriate tool to use based off the user’s question. Each tool will be capable of calling on any of the other tools available to the LLM, resulting in a collaborative pipeline where tools can pass data between other tools until enough data is received to generate an answer to the user’s question. Using a locally deployed, open-source, LLM, the NASA OCARI team, in collaboration with Collins Aerospace, will implement a prototype application that will bridge knowledge across multiple sources to assist System Engineers (SEs) with requirements discovery and tracing, research question and use case identification, and assumption validation. Such a system will also allow SEs to more easily, and intuitively, explore the AAM ecosystem, ultimately improving the efficiency and effectiveness of the SE's research and decision-making processes surrounding ConOps development and validation. In this session, our team will provide a video demonstration of our new prototype architecture in action. We will also present an overview of our prototype system architecture and talk about its advantages over traditional LLM deployments along with how those advantages can provide additional value to the field of System Engineering.
Machine learning technology spans many areas and today plays a significant role in addressing a wide range of problems in critical domains,i.e., healthcare, autonomous driving, finance, manufacturing, cybersecurity,etc. Metamorphic testing (MT) is considered a simple but very powerful approach in testing such computationally complex systems for which either an oracle is not available or is available but difficult to apply. Conventional metamorphic testing techniques have certain limitations in verifying deep learning-based models (i.e., convolutional neural networks (CNNs)) that have a stochastic nature (because of randomly initializing the network weights) in their training. In this article, we attempt to address this problem by using a statistical metamorphic testing (SMT) technique that does not require software testers to worry about fixing the random seeds (to get deterministic results) to verify the metamorphic relations (MRs). We propose seven MRs combined with different statistical methods to statistically verify whether the program under test adheres to the relation(s) specified in the MR(s). We further use mutation testing techniques to show the usefulness of the proposed approach in the healthcare space and test two CNN-based deep learning models (used for pneumonia detection among patients). The empirical results show that our proposed approach uncovers 85.71% of the implementation faults in the classifiers under test (CUT). Furthermore, we also propose an MRs minimization algorithm for the CUT, thus saving computational costs and organizational testing resources.
A modeling method to calculate the intervals of uncertainty for parameters found by identification is described. The region of confidence and the general approach to the calculation of these intervals are discussed. The general subprograms for determination of dimensions are described. They provide the organizational charts for the subprograms, the tests carried out and the listings of the different subprograms.