Search NASA⌕ Search

SEARCH · Search NASA

Results for “Test Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Autonomous System Inference, Trojan, and Adversarial Reprogramming Attack and Defense (Final)

In the world of ever-advancing technology, Autonomous Systems (AS) find extensive application, bolstering functionalities of critical infrastructures such as nuclear power plants. These systems, however, are increasingly becoming a target for nefarious activities, namely through inference attacks, trojan attacks, and adversarial reprogramming. This paper delves into a comprehensive exploration of machine learning (ML)-driven autonomous control systems within advanced nuclear reactor designs, revealing the vulnerabilities and proposing strategies for defense against potential cyber-attacks. Advanced cyber-attacks against critical infrastructure and the energy sector are becoming more common. With the invention of autonomous control systems (ACS) within advanced nuclear reactor designs, system designers, reactor operators, and regulators must consider cybersecurity during the design and operational phases. This article provides a cyber threat assessment of machine learning (ML)- based digital twinning (DT) technologies in the context of advanced reactor ACS. A cyber-physical testbed was created to emulate nuclear reactor digital instrumentation and controls (I&C) and act as a basis for the ACS. The ACS was designed as two plant-level DTs predicting reactor malfunctions and determining control actions and two component-level DTs responsible for classifying component states and forecasting component inputs and outputs (I/O). Two duplicate ACS designs– one using a traditional ML framework and one using an automated ML (AutoML) framework– were created and tested against cyber-attacks on training data, real-time process data, and ML model architectures to determine their respective qualitative cyber-risk in terms of likelihood and impact. Both frameworks showed similar cyber-resilience against training, real-time, and ML architecture attacks, proving that neither is inherently more secure. Recommended safeguard and security measures are posed to system designers, reactor operators, and regulators to maintain the cybersecurity of ML-based DT technologies such as ACS, prompting a holistic view of shared responsibility for maintaining cyber-secure ML-based systems. As global reliance on generation III reactors begins to be critically assessed, the evolution towards advanced reactor systems utilizing digital instrumentation and controls (I&C) becomes not merely preferable, but essential. The integration of semi and fully autonomous control systems (ACS), powered by digital I&C and machine learning (ML)-based digital twinning (DT) technologies, emerges as a potent strategy to mitigate operations and maintenance costs, thereby enhancing the economic feasibility of novel reactor designs. However, with a staggering 500% and 380% increase in cyber-attacks reported against the energy sector by the United States Department of Energy (DoE) and the European Union respectively, a surge in cyber vulnerabilities specifically targeting the nuclear industry has been 2 markedly observed. Notable incidents, such as the W32.Ramnit spyware infiltration at the Gundremmingen nuclear power plant in Germany and the Dtrack spyware intrusion at the Kudankulam nuclear power plant in India, while not directly compromising core industrial control systems (ICS), underscore a compelling necessity to fortify cybersecurity protocols in safeguarding reactor systems against increasingly adept digital adversaries. In light of this, our investigation extends beyond conventional cybersecurity parameters, diving into the intricate web of potential vulnerabilities woven into ML-based DTs and ACS in advanced reactor systems. A crafted cyber-physical testbed and preliminary ACS were devised to act as a mirror, reflecting potential configurations of advanced reactor control designs. Moreover, this study is intertwined with a scrutinization of ML models, developed either through conventional, manually tuned methodologies or via automated means through AutoML, probing into their cyber-risk profiles within operational technology (OT) environments. Expanding on this, two distinct ACS blueprints were forged – one navigating through the corridors of traditional ML and the other traversing the path of AutoML – in an effort to holistically encapsulate the considerations pivotal to ML-based DT control system design. Employing the SANS Institute Industrial Control System (ICS) Kill Chain and the MITRE ATT&CK Tactics, Techniques, and Procedures (TTP) framework, a structured analysis was conducted, launching three targeted attacks against the training dataset, real-time dataset, and ML models, therein dissecting the potential cyber-attack implications against both ML frameworks within an ACS milieu. It is essential to note that three distinct categories of attacks were conducted against both ACS configurations, each encompassing three distinct ML-based DTs, cumulating in a total of 18 varied attacks. This exploration extends into the realms of Autonomous System Inference, Trojan, and Adversarial Reprogramming Attack and Defense, unraveling vulnerabilities, and opportunities for fortified defenses against such intrusions, particularly where ML-driven technologies, and by extension, ACS, are deployed. Final recommendations, articulated through a lens of security, safeguard, and implementation considerations, are presented for both traditional and AutoML models, anchoring upon the existing knowledge landscape and ML-based DT modeling for ACS, and are offered as a beacon to guide the nuclear industry through the intricate cybersecurity challenges that lie ahead.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Rapid Prototyping of an Aircraft Model in an Object-Oriented Simulation

A team was created to participate in the Mars Scout Opportunity. Trade studies determined that an aircraft provided the best opportunity to complete the science objectives of the team. A high fidelity six degree of freedom flight simulation was required to provide credible evidence that the aircraft design fulfilled mission objectives and to support the aircraft design process by providing performance evaluations. The team created the simulation using the Langley Standard Real-Time Simulation in C++ (LaSRS++) application framework. A rapid prototyping approach was necessary because the team had only three months to both develop the aircraft simulation model and evaluate aircraft performance as the design and mission parameters matured. The design of LaSRS++ enabled rapid-prototyping in several ways. First, the framework allowed component models to be designed, implemented, unit-tested, and integrated quickly. Next, the framework provides a highly reusable infrastructure that allowed developers to maximize code reuse while concentrating on aircraft and mission specific features. Finally, the framework reduces risk by providing reusable components that allow developers to build a quality product with a compressed testing cycle that relies heavily on unit testing of new components.

Kenney, P. Sean↗

New vertical cryostat for the high field superconducting magnet test station at CERN

In the framework of the R and D program for new superconducting magnets for the Large Hadron Collider accelerator upgrades, CERN is building a new vertical test station to test high field superconducting magnets of unprecedented large size. This facility will allow testing of magnets by vertical insertion in a pressurized liquid helium bath, cooled to a controlled temperature between 4.2 K and 1.9 K. The dimensions of the cryostat will allow testing magnets of up to 2.5 m in length with a maximum diameter of 1.5 m and a mass of 15 tons. To allow for a faster insertion and removal of the magnets and reducing the risk of helium leaks, all cryogenics supply lines are foreseen to remain permanently connected to the cryostat. A specifically designed 100 W heat exchanger is integrated in the cryostat helium vessel for a controlled cooling of the magnet from 4.2 K down to 1.9 K in a 3 m{sup 3} helium bath. This paper describes the cryostat and its main functions, focusing on features specifically developed for this project. The status of the construction and the plans for assembly and installation at CERN are also presented.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Feasibility Study on Implementing a Staggered-Grid Finite Volume Method for System Analysis Code Development Under the MOOSE Framework

Here, this work summarizes a feasibility study on testing numerical algorithms that are suitable and efficient for advanced system analysis code development under the mutli-physics framework, MOOSE. The key to the test bed is the implementation of high-order one-dimensional staggered-grid finite volume method (SG-FVM), and its direct interaction with the linear/nonlinear solver, PETSc. The test bed utilized a more flexible code structure to enable the finite volume method implementation and direct interacting with the solver package, instead of using the natively supported finite element method by the framework. Using a suite of selected test problems with different problem sizes and levels of complexity, the implemented SG-FVM demonstrated superior performance improvement against a direct finite element method implementation through MOOSE. On two computer systems, the speedup was observed to be significant, with at least one order of magnitude of solving time reduction. For a complex reactor model, transient simulation was performed using the newly developed finite volume method code, the results of which agree very well with the reference results from the finite element method code. Overall, this study demonstrates a successful feasibility study on the proposed numerical algorithms and software structure to support advanced system analysis tool development.

MOOSE↗

Quantifying Climate and Catchment Control on Hydrological Drought in the Continental United States

The evolution of hydrological drought events is a result of complex (nonlinear) interactions between climate and catchment processes. To investigate such nonlinear relationship, we integrated a machine learning modeling framework based on the random forest (RF) algorithms with an interpretation framework to quantify the role of climate and catchment controls on hydrological drought. More particularly, our framework interprets a built RF machine-learning model to identify dominant variables and visualize their functional dependence and interaction effects on hydrological drought characteristics utilizing concepts of minimal depth, interactive depth, and partial dependence. We test our proposed modeling framework based on a set of 652 continental United States catchments with minimal human interference for a period of 1979–2010. Application of this framework indicated presence of three distinct drought regimes, which includes, Regime 1: droughts with longer duration, less frequent and lesser intensity; Regime 2: droughts with moderate duration, moderate frequency, and moderate intensity; and Regime 3: droughts with shorter duration, more frequent, and more intense. RF algorithm was able to accurately model the drought characteristics (intensity, duration, and number of events) for all the three drought regimes as a function of selected variables. It was observed that the type of dominant variables as well as their nonlinear functional relationship with hydrological droughts characteristics can vary between three selected regimes. Finally, our interpretation framework indicated that catchment characteristics have a significant role in controlling the hydrologic drought for catchments (regime 1), whereas both climate and catchment characteristics control hydrological drought in regimes 2 and 3.

54 ENVIRONMENTAL SCIENCES↗

Playing with DIRT: Building the Framework for a Comprehensive In-Situ Soil Materials Testing Database

Long-term, sustainable planetary exploration will require the ability to "live off the land," relying on In-Situ Resource Utilization (ISRU) and In-Situ Construction as core capabilities. Reduction of both risk and launch mass for lunar construction will require evaluating and comparing regolith materials for use as feedstocks suitable for in-situ beneficiation and fabrication of building components. These material assessment capabilities will inform lunar infrastructure design decisions, with co-benefits for terrestrial construction using in-situ materials. Current planetary construction technology development relies on lunar mapping and orbital data, Apollo-era sample analyses, current simulant inventories, and tests conducted using analog site soils. Requirements for lunar infrastructure design decisions and construction systems will be determined based on specific environmental conditions, mission architectures, and the materials available within traverse range of lunar feedstock processing depots. Critical for success is the capability to identify, evaluate and make effective use of a wide range of materials as they are found in-situ on the lunar surface. This paper discusses the development of a digital repository for data on soil and regolith properties, beginning with their structural performance in both cementitious and non-cementitious building material formulations. The Database for In-situ Resource Testing (DIRT) compiles a catalog of raw materials, additives, and formulations, with notations pertaining to material sources and preparation techniques entered via a web-based user interface. Design of consistent data schemas for site-sourced materials evaluation will facilitate linkage with relevant terrestrial and planetary materials databases while enabling guided data input via templates for participation by broader groups of collaborators. Results of these analyses are compiled in a centralized repository to generate insights applicable for regolith resources and landing sites yet to be precisely defined. Collection of thorough records of material characteristics, applications and performance will support innovative construction solutions not only for space infrastructure but for sustainable, resilient design in the terrestrial built environment.

Sarah Joey Seitz↗

NCCS Regression Test Harness

The NCCS Regression Test Harness is a software package that provides a framework to perform regression and acceptance testing on NCCS High Performance Computers. The package is written in Python and has only the dependency of a Subversion repository to store the regression tests.

Tharrington, Arnold N.↗

Optimal Nested Test Plan for Combinatorial Quantitative Group Testing

We consider the quantitative group testing problem where the objective is to identify defective items in a given population based on results of tests performed on subsets of the population. Under the quantitative group testing model, the result of each test reveals the number of defective items in the tested group. The minimum number of tests achievable by nested test plans was established by Aigner and Schughart in 1985 within a minimax framework. Furthermore, the optimal nested test plan offering this performance, however, was not obtained. In this paper, we establish the optimal nested test plan in a closed form. This optimal nested test plan is also order optimal among all test plans as the population size approaches infinity. Using heavy-hitter detection as a case study, we show via simulation examples orders of magnitude improvement of the group testing approach over two prevailing sampling-based approaches in detection accuracy and counter consumption. Other applications include anomaly detection and wideband spectrum sensing in cognitive radio systems.

97 MATHEMATICS AND COMPUTING↗

Particle Filter Based Inference Testing

The primary intent of PAR-FIT (Particle Filter based Inference Testing) is to provide hard inductive evidence that a machine learning model is capable and proven for an individual test input. By examining training data used to form the underlying model functional correlation, an estimate of the reliability that a model will make the correct prediction can be made. The Sequential Probability Ratio Test is used to derive a qualitative evaluation for reliability based on hypothesis testing. The PAR-FIT framework achieves this by implementing a particle filter and the sequential probability ratio test algorithms on the machine learning model training data to determine relevancy of new individual test samples to the training dataset. The kernel function evaluates the local proximity and density of training data used to derive a prediction outcome. Particles are used to probabilistically determine which training data to evaluate for proximity. For test samples that are within a close proximity to and surrounded by multiple training data points, the evaluated reliability of the prediction is high. For test samples that are anomalies not represented by the training dataset, in low density data clusters, or are far from existing data points, the evaluated reliability is low as insufficient training evidence exists to suggest the model is capable of making the correct prediction. Sequential Probability Ratio Test is further used to determine when a hypothesis on whether a signal can be rejected or accepted for use. The ratio test collects sequence information from the particle filter to test whether the signal is anomalous or normal via hypothesis testing of the underlying distributions.

Chen, Edward [Idaho National Laboratory (INL), Ida↗

Effect of Entropic Constraints on the Thermodynamics of Molecular Adsorption in Nano‐Porous Materials

Abstract Gas separation is a critical industrial process that consumes a significant amount of energy due to the widely used techniques that are currently employed. Adsorptive materials—such as metal–organic frameworks (MOFs)—show promise as an energy‐efficient alternative. Of particular current interest are novel, temperature‐dependent separation processes in MOFs, such as the recently reported separation of ternary isomeric hydrocarbon mixtures within one and the same material. However, the mechanisms of these highly desirable separations remain poorly understood. Herein, through a combination of ab initio simulations and statistical mechanics, it is shown that the temperature dependence is the result of a constraint on the guest molecule's entropic degrees of freedom when loaded into the MOF, caused by the fortuitous tight fitting of the guest inside the pore. While the framework applies to all molecular adsorption in porous media, it is essential for the description of large molecules in small pores, which is demonstrated here using the separation of C6 isomers in Ca(H 2 tcpb) as a test case. The developed framework and analysis not only reveal the reason why separation occurs but also predict the temperatures at which it takes place, thus opening the door to newly designed MOFs with tailor‐made precision.

Chemistry↗

On the estimation of boundary layer heights: a machine learning approach

Abstract. The planetary boundary layer height (zi) is a key parameter used in atmospheric models for estimating the exchange of heat, momentum, and moisture between the surface and the free troposphere. Near-surface atmospheric and subsurface properties (such as soil temperature, relative humidity, etc.) are known to have an impact on zi. Nevertheless, precise relationships between these surface properties and zi are less well known and not easily discernible from the multi-year dataset. Machine learning approaches, such as random forest (RF), which use a multi-regression framework, help to decipher some of the physical processes linking surface-based characteristics to zi. In this study, a 4-year dataset from 2016 to 2019 at the Southern Great Plains site is used to develop and test a machine learning framework for estimating zi. Parameters derived from Doppler lidars are used in combination with over 20 different surface meteorological measurements as inputs to a RF model. The model is trained using radiosonde-derived zi values spanning the period from 2016 through 2018 and then evaluated using data from 2019. Results from 2019 showed significantly better agreement with the radiosonde compared to estimates derived from a thresholding technique using Doppler lidars only. Noteworthy improvements in daytime zi estimates were observed using the RF model, with a 50 % improvement in mean absolute error and an R2 of greater than 85 % compared to the Tucker method zi. We also explore the effect of zi uncertainty on convective velocity scaling and present preliminary comparisons between the RF model and zi estimates derived from atmospheric models.

54 ENVIRONMENTAL SCIENCES↗

Multidisciplinary Optimization Branch Experience Using iSIGHT Software

The Multidisciplinary Optimization (MDO) Branch at NASA Langley Research Center is investigating frameworks for supporting multidisciplinary analysis and optimization research. An optimization framework call improve the design process while reducing time and costs. A framework provides software and system services to integrate computational tasks and allows the researcher to concentrate more on the application and less on the programming details. A framework also provides a common working environment and a full range of optimization tools, and so increases the productivity of multidisciplinary research teams. Finally, a framework enables staff members to develop applications for use by disciplinary experts in other organizations. Since the release of version 4.0, the MDO Branch has gained experience with the iSIGHT framework developed by Engineous Software, Inc. This paper describes experiences with four aerospace applications: (1) reusable launch vehicle sizing, (2) aerospike nozzle design, (3) low-noise rotorcraft trajectories, and (4) acoustic liner design. All applications have been successfully tested using the iSIGHT framework, except for the aerospike nozzle problem, which is in progress. Brief overviews of each problem are provided. The problem descriptions include the number and type of disciplinary codes, as well as all estimate of the multidisciplinary analysis execution time. In addition, the optimization methods, objective functions, design variables, and design constraints are described for each problem. Discussions on the experience gained and lessons learned are provided for each problem. These discussions include the advantages and disadvantages of using the iSIGHT framework for each case as well as the ease of use of various advanced features. Potential areas of improvement are identified.

Padula, S. L.↗

Analysis of Complex Valve and Feed Systems

A numerical framework for analysis of complex valve systems supports testing of propulsive systems by simulating key valve and control system components in the test loop. In particular, it is designed to enhance the analysis capability in terms of identifying system transients and quantifying the valve response to these transients. This system has analysis capability for simulating valve motion in complex systems operating in diverse flow regimes ranging from compressible gases to cryogenic liquids. A key feature is the hybrid, unstructured framework with sub-models for grid movement and phase change including cryogenic cavitations. The multi-element unstructured framework offers improved predictions of valve performance characteristics under steady conditions for structurally complex valves such as pressure regulator valve. Unsteady simulations of valve motion using this computational approach have been carried out for various valves in operation at Stennis Space Center such as the split-body valve and the 10-in. (approx.25.4-cm) LOX (liquid oxygen) valve and the 4-in. (approx.10 cm) Y-pattern valve (liquid nitrogen). Such simulations make use of variable grid topologies, thereby permitting solution accuracy and resolving important flow physics in the seat region of the moving valve. An advantage to this software includes possible reduction in testing costs incurred due to disruptions relating to unexpected flow transients or functioning of valve/flow control systems. Prediction of the flow anomalies leading to system vibrations, flow resonance, and valve stall can help in valve scheduling and significantly reduce the need for activation tests. This framework has been evaluated for its ability to predict performance metrics like flow coefficient for cavitating venturis and valve coefficient curves, and could be a valuable tool in predicting and understanding anomalous behavior of system components at rocket propulsion testing and design sites.

Ahuja, Vineet↗

An Automated DAKOTA and VULCAN-CFD Framework with Application to Supersonic Facility Nozzle Flowpath Optimization

Removing human interaction from design processes by using automation may lead to gains in both productivity and design precision. This memorandum describes efforts to incorporate high fidelity numerical analysis tools into an automated framework and applying that framework to applications of practical interest. The purpose of this effort was to integrate VULCAN-CFD into an automated, DAKOTA-enabled framework with a proof-of-concept application being the optimization of supersonic test facility nozzles. It was shown that the optimization framework could be deployed on a high performance computing cluster with the flow of information handled effectively to guide the optimization process. Furthermore, the application of the framework to supersonic test facility nozzle flowpath design and optimization was demonstrated using multiple optimization algorithms.

Axdahl, Erik L.↗

SCCS System SW Development

The original project to be completed, GenSim, was a Generic Simulator for Application Control Language scripts. This would mock the systems on which the scripts were meant to be run on so that you could run tests without access to the machine. Soon after the project was started, a different project took priority. Unit testing the Application Services Framework code became the focus of the team because of approaching deadlines. Communication was very important to ensure that code wasn't being duplicated and to keep the team up to date with what mock files are available to them. Because of this, daily meetings were conducted until the testing was complete.

computer programming↗

OLCF Test Harness

Acceptance and regression testing of a High Performance Computing (HPC) system requires an automated and reproducible framework and tool for running and logging results. Manually running tests across a system is labor intensive and prone to reproducibility errors. The OLCF Test Harness (OTH) provides a framework in which to document required tests for a HPC system. The OTH then provides tools to execute and log results of these tests in an automated fashion.

Dietz, Dan [Oak Ridge National Laboratory (ORNL), ↗