Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 793 records · Page 44

Implementing Software Safety in the NASA Environment

Until recently, NASA did not consider allowing computers total control of flight systems. Human operators, via hardware, have constituted the ultimate safety control. In an attempt to reduce costs, NASA has come to rely more and more heavily on computers and software to control space missions. (For example. software is now planned to control most of the operational functions of the International Space Station.) Thus the need for systematic software safety programs has become crucial for mission success. Concurrent engineering principles dictate that safety should be designed into software up front, not tested into the software after the fact. 'Cost of Quality' studies have statistics and metrics to prove the value of building quality and safety into the development cycle. Unfortunately, most software engineers are not familiar with designing for safety, and most safety engineers are not software experts. Software written to specifications which have not been safety analyzed is a major source of computer related accidents. Safer software is achieved step by step throughout the system and software life cycle. It is a process that includes requirements definition, hazard analyses, formal software inspections, safety analyses, testing, and maintenance. The greatest emphasis is placed on clearly and completely defining system and software requirements, including safety and reliability requirements. Unfortunately, development and review of requirements are the weakest link in the process. While some of the more academic methods, e.g. mathematical models, may help bring about safer software, this paper proposes the use of currently approved software methodologies, and sound software and assurance practices to show how, to a large degree, safety can be designed into software from the start. NASA's approach today is to first conduct a preliminary system hazard analysis (PHA) during the concept and planning phase of a project. This determines the overall hazard potential of the system to be built. Shortly thereafter, as the system requirements are being defined, the second iteration of hazard analyses takes place, the systems hazard analysis (SHA). During the systems requirements phase, decisions are made as to what functions of the system will be the responsibility of software. This is the most critical time to affect the safety of the software. From this point, software safety analyses as well as software engineering practices are the main focus for assuring safe software. While many of the steps proposed in this paper seem like just sound engineering practices, they are the best technical and most cost effective means to assure safe software within a safe system.

Wetherholt, Martha S.↗

Effects of size, geometry, and testing temperature on additively manufactured Ti-6Al-4V titanium alloy

This article presents a comprehensive study concerning size and geometry effect on the ambient and high-temperature mechanical behavior of additively manufactured laser powder bed fusion Ti-6Al-4V alloy. Key mechanical property metrics are presented, including strain hardening rate, yield strength, and Young’s modulus as a function of thickness and testing temperature (ambient, 250 °C, and 450 °C). Here, the effect of specimen size on manufacturing-induced microstructural feature formation is demonstrated and discussed. A detailed analysis regarding Young’s modulus, yield strength, and strain hardening rate variation at elevated temperatures is also presented. Size effect and temperature-sensitive deformation mechanisms are linked to the underlying microstructural deformation mechanisms activated at respective temperatures. Counter-intuitively, this study determined that irrespective of the geometry, strain hardening rates increased as temperature increased. An in-depth microstructural examination is presented to explain the stated observation. The texture of Interrupted tensile test samples examined for ambient, 250 °C, and 450 °C post-yield was observed to be unchanged, indicating the strain hardening behavior was not texture dependent. Schmid factor analysis, coupled with experimental findings from previous work, was implemented to generate a hypothesis. This hypothesis suggests that the drop in critical resolved shear stress for basal and pyramidal slip systems, as temperature increases, paired with the high dislocation density of laser powder bed fusion Ti-6Al-4V, leads to dislocation entanglement and results in increased strain hardening at elevated temperatures.

36 MATERIALS SCIENCE↗

Development of Hydrothermal Liquefaction and Upgrading Technologies for Lipid-Extracted Algae Conversion to Liquid Fuels

Bench-scale tests were performed for lipid-extracted microalgae (LEA) conversion to liquid fuels via hydrotreating liquefaction (HTL) and upgrading processes. Process simulation and economic analysis for a large-scale LEA HTL and upgrading system were developed based on the best available test results. The system assumes an LEA feed rate of 608 dry metric ton/day and that the feedstock is converted to a crude HTL bio-oil and further upgraded via hydrotreating and hydrocracking to produce liquid hydrocarbon fuels, mainly alkanes. Performance and cost results demonstrate that HTL would be an effective option to convert LEA to liquid fuel. The liquid fuels annual yield was estimated to be 26.9 million gallon gasoline-equivalent and the overall energy efficiency at higher heating value basis was estimated to be 69.5%. The minimum fuel selling price (MFSP) was estimated to be $0.75/L with LEA feedstock price at $33.1 metric ton at dry basis and 10% internal rate of return. A sensitivity analysis indicated that the largest effects to production cost would come from the final products yields and the upgrading equipments cost. The impact of plant scale on MFSP was also investigated.

Microalgae, Lipid extracted algae, Hydrothermal li↗

Charged particle in higher dimensional weakly charged rotating black hole spacetime

We study charged particle motion in weakly charged higher dimensional black holes. To describe the electromagnetic field we use a test field approximation and the higher dimensional Kerr-NUT-(A)dS metric as a background geometry. It is shown that for a special configuration of the electromagnetic field, the equations of motion of charged particles are completely integrable. The vector potential of such a field is proportional to one of the Killing vectors (called a primary Killing vector) from the 'Killing tower' of symmetry generating objects which exists in the background geometry. A free constant in the definition of the adopted electromagnetic potential is proportional to the electric charge of the higher dimensional black hole. The full set of independent conserved quantities in involution is found. We demonstrate that Hamilton-Jacobi equations are separable, as is the corresponding Klein-Gordon equation and its symmetry operators.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

TRIM: AI Guided Random Number Generation for Resource-Constrained IoT Systems

Random numbers often serve as the backbone for many security solutions in diverse domains such as cryptography, side channel leakage prevention, and moving target defense. However, generating true random numbers requires a physical source of entropy (e.g. hardware, quantum, environmental phenomenon) making it difficult to realize at a large scale and at a low cost. On the flip side, pseudorandom number generators (easy to implement) following a specific distribution (e.g. Gaussian) can be easily compromised given a sufficient amount of traces. In this work, we have developed a machine learning-guided generative approach that can be used to create portable, resource-efficient, and cost-effective random number generators with high throughput and true randomness characteristics. We implement the proposed approach as a highly parameterized framework and perform extensive evaluation for different settings. The framework was able to learn from true random sources such as irrational numbers and environmental audio noise and imitate those sources towards generating new good quality random numbers on demand. We have generated more than 1 billion bits and observed robust performance in terms of true randomness metrics obtained from NIST SP 800-22 and FIPS 140-1 randomness test suites achieving a throughput of up to 142.85 Mbps. Compared to the state-of-the-art (SOTA) technique, the iso-cost setup of our framework can achieve more than 500 Mbps in a distributed setting. We have evaluated the efficacy of running the true randomness imitation AI models on target edge devices such as Raspberry Pi 4 (Model B), Nvidia Jetson Nano, Nvidia Jetson Orin Nano and Nvidia Jetson Xavier. We have also looked at the security of the TRIM framework itself against different adversarial threat models.

Cybersecurity↗

Enabling Fairness in Distributed PV Curtailment for Voltage Regulation: A Comparative Analysis

The growing adoption of photovoltaic systems in power distribution networks has yielded numerous advantages, but it has also introduced challenges, particularly in managing overvoltage issues. Uncoordinated photovoltaic integration can lead to voltage rise beyond acceptable levels. Curtailing active power is an effective method for addressing overvoltage issues in power distribution systems. Nevertheless, it is essential to distribute the curtailment fairly among the resources to maintain fairness and achieve a well-balanced utilization of renewable energy. This paper presents a comparative study of different fairness schemes for active power curtailment in photovoltaic integration systems. Three different curtailment methods are selected for demonstration: i) proportional, ii) egalitarian, and iii) financial. The study involves evaluating the curtailment schemes based on their ability to distribute the curtailed power among the photovoltaic systems in a justifiable manner, focusing on their formulation and distributed solution. A detailed performance comparison is carried out to highlight the cost of catering for fairness schemes in photovoltaic curtailment using relevant metrics, where simulations are carried out using a modified IEEE 123-bus test case.

Mukherjee, Monish↗

An Evaluation of the Flywheel Potential for Providing Regulation Service in California

Flywheels can provide regulation and frequency response services to the power grids. This study presents the technical characteristics, modeling approach, methodologies, and results for providing regulation services in the California Independent System Operator (CAISO) market. Breakeven cost analyses were developed for two cases: 1) flywheel provides the regulation service alone; and 2) flywheel provides the regulation service together with a hydro power plant. For both cases, we evaluated two payment methods: pay-by-energy and pay-by-capacity. Based on the results of the technical and cost analyses, the opportunities for the flywheel providing regulation services are discussed; field test results for the flywheel’s physical characteristics are presented; and performance metrics of the flywheel to provide the regulation services are suggested.

25 ENERGY STORAGE↗

VULCON: A System for Vulnerability Prioritization, Mitigation, and Management

Vulnerability remediation is a critical task in operational software and network security management. In this paper, an effective vulnerability management strategy, called VULCON (VULnerability CONtrol), is developed and evaluated. The strategy is based on two fundamental performance metrics: i). Time-to-Vulnerability Remediation (TVR) and; ii). Total Vulnerability Exposure (TVE). VULCON takes as input real vulnerability scan reports, metadata about the discovered vulnerabilities, asset criticality, and personnel resources. VULCON uses a mixed integer multi-objective optimization algorithm to prioritize vulnerabilities for patching, such that the above performance metrics are optimized subject to the given resource constraints. VULCON has been tested on multiple months of real scan data from a Cyber-Security Operations Center (CSOC). Results indicate an overall Total Vulnerability Exposure reduction of 8.97\% when VULCON optimizes a realistic security analyst workforce's effort. Additionally, it is demonstrated that VULCON can determine monthly resources required to maintain a target TVE score. As such, VULCON provides valuable operational guidance for improving vulnerability response processes in CSOCs.

Farris, Katheryn A.↗

SRF Cavity Instability Detection with Machine Learning at CEBAF

During the operation of CEBAF, one or more unstable superconducting radio-frequency (SRF) cavities often cause beam loss trips while the unstable cavities themselves do not necessarily trip off. Identifying an unstable cavity out of the hundreds of cavities installed at CEBAF is difficult and time-consuming. The present RF controls for the legacy cavities report at only 1 Hz, which is too slow to detect fast transient instabilities. A fast data acquisition system for the legacy SRF cavities is being developed which samples and reports at 5 kHz to allow for detection of transients. A prototype chassis has been installed and tested in CEBAF. An autoencoder based machine learning model is being developed to identify anomalous SRF cavity behavior. The model is presently being trained on the slow (1 Hz) data that is currently available, and a separate model will be developed and trained using the fast (5 kHz) DAQ data once it becomes available. This paper will discuss the present status of the new fast data acquisition system and results of testing the prototype chassis. This paper will also detail the initial performance metrics of the autoencoder model.

Carpenter, A.↗

Ford Thermoelectric HVAC Project (Final Report)

Current light-duty vehicles provide passenger thermal comfort primarily through the use of a centralized HVAC (Heating, Ventilation, and Air Conditioning) unit that distributes conditioned air to vent locations throughout the vehicle. A substantial portion of the air energy content is spent conditioning various elements of the vehicle's interior structures. This is not an overwhelming issue in today’s vehicles, but as pressure builds to improve powertrain efficiency, the energy available to provide occupant comfort will decrease. Thermoelectric devices were identified by the US Department of Energy as one method to potentially reduce HVAC-related energy consumption in vehicles. The objective of the Ford Thermoelectric HVAC Project was to identify a technical and business approach to accelerate research and development of light-duty automotive thermoelectric HVAC technology. A zonal HVAC system was developed to provide thermal comfort to each occupant, with the objective of reducing the overall power consumed by the HVAC system compared with a baseline system. The system was designed to provide a basis for a down-sized central HVAC unit, while providing equivalent comfort and equivalent heating/cooling time-to-comfort at startup, and reduce the energy consumed to provide these functions. Project success was evaluated using occupant comfort and energy consumption metrics, based on a novel thermal comfort-modeling approach, and empirical testing. The ability to produce and market the thermoelectric devices developed in this program was assessed by a business analysis that outlined key steps required for thermoelectric HVAC devices to come to market. Key hardware and controls attribute requirements were identified to determine how they contribute to the commercialization potential of the technology. Final testing and analysis showed that use of thermal comfort and zonal design criteria reduced the energy consumption of the climate control system in both heating and cooling tests, and were significantly better in low vehicle occupancy cases where zonal strategies could be employed. Thermoelectric devices were utilized in the vehicle design to support achieving the technical performance goals of the project. It was found that these devices need additional development in order to address commercial issues, due to their cost and performance limitations.

99 GENERAL AND MISCELLANEOUS↗

Physical Security Model Development of an Electrochemical Facility

Nuclear facilities in the U.S. and around the world face increasing challenges in meeting evolving physical security requirements while keeping costs reasonable. The addition of security features after a facility has been designed and without attention to optimization (the approach of the past) can easily lead to cost overruns. Instead, security should be considered at the beginning of the design process in order to provide robust, yet efficient physical security designs. The purpose of this work is to demonstrate how modeling and simulation can be used to optimize the design of physical protection systems. A suite of tools, including Scribe3D and Blender, were used to model a generic electrochemical reprocessing facility. Physical protection elements such as sensors, portal monitors, barriers, and guard forces were added to the model based on best practices for physical security. Two theft scenarios (an outsider attack and insider diversion) as well as a sabotage scenario were examined in order to optimize the security design. Security metrics are presented. This work fits into a larger Virtual Facility Distributed Test Bed 2020 Milestone in the Material Protection, Accounting, and Control Technologies (MPACT) program through the Department of Energy (DOE). The purpose of the milestone is to demonstrate how a series of experimental and modeling capabilities across the DOE complex provide the capabilities to demonstrate complete Safeguards and Security by Design (SSBD) for nuclear facilities.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

SRF Cavity Instability Detection with Machine Learning at CEBAF

During the operation of CEBAF, one or more unstable superconducting radio-frequency (SRF) cavities often cause beam loss trips while the unstable cavities themselves do not necessarily trip off. Identifying an unstable cavity out of the hundreds of cavities installed at CEBAF is difficult and time-consuming. The present RF controls for the legacy cavities report at only 1 Hz, which is too slow to detect fast transient instabilities. A fast data acquisition system for the legacy SRF cavities is being developed which samples and reports at 5 kHz to allow for detection of transients. A prototype chassis has been installed and tested in CEBAF. An autoencoder based machine learning model is being developed to identify anomalous SRF cavity behavior. The model is presently being trained on the slow (1 Hz) data that is currently available, and a separate model will be developed and trained using the fast (5 kHz) DAQ data once it becomes available. This paper will discuss the present status of the new fast data acquisition system and results of testing the prototype chassis. This paper will also detail the initial performance metrics of the autoencoder model.

Carpenter, A.↗

AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)

Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.

97 MATHEMATICS AND COMPUTING↗

An Evaluation of Co-Simulation for Modeling Coupled Natural Gas and Electricity Networks

Reliance on natural gas for power generation has increased the coupling between gas and power networks. While this coupling can bring operational and economic benefits, it can also yield challenges, as the constraints in one system can impact the other. Co-simulation can capture the constraints and interactions between these systems, but so far, there has been limited comparison of co-simulation results to those of an integrated model. In this work, we develop a new co-simulation framework using the HELICS platform and the SAInt tool for modeling transient gas and AC optimal power flow. We evaluate this co-simulation framework against a fully integrated version of the SAInt power and gas simulators, thus providing a benchmarking of the co-simulation approach. We compare results across the two approaches for two test networks and a network representing the Belgian power and gas networks, testing both normal operating conditions and cases with compressor disruptions. In each of the cases tested, we find nearly identical results from the two approaches across various metrics of interest, such as nodal pressure, gas flow rates, and active power generation. This alignment suggests that co-simulation can yield comparable results to fully integrated models for modeling coupled gas and electricity networks.

03 NATURAL GAS↗

Sensitivities of subgrid-scale physics schemes, meteorological forcing, and topographic radiation in atmosphere-through-bedrock integrated process models: a case study in the Upper Colorado River basin

Abstract. Mountain hydrology is controlled by interacting processes extending from the atmosphere through the bedrock. Integrated process models (IPMs), one of the main tools needed to interpret observations and refine conceptual models of the mountainous water cycle, require meteorological forcing that simulates the atmospheric process to predict hydroclimate then subsequently impacts surface–subsurface hydrology. Complex terrain and extreme spatial heterogeneity in mountainous environments drive uncertainty in several key considerations in IPM configurations and require further quantification and sensitivity analyses. Here, we present an IPM using the Weather Research and Forecasting (WRF) model which forces an integrated hydrologic model, ParFlow-CLM, implemented over a domain centered over the East River watershed (ERW), located in the Upper Colorado River basin (UCRB). The ERW is a heavily instrumented 300 km2 region in the headwaters of the UCRB near Crested Butte, CO, with a growing atmosphere-through-bedrock observation network. Through a series of experiments in the water year 2019 (WY19), we use four meteorological forcings derived from commonly used reanalysis datasets, three subgrid-scale physics scheme configurations in WRF, and two terrain shading options within WRF to test the relative importance of these experimental design choices for key hydrometeorological metrics including precipitation and snowpack, as well as evapotranspiration, groundwater storage, and discharge simulated by the ParFlow-CLM. Our hypothesis is that uncertainty from synoptic-scale forcings produces a much larger spread in surface–subsurface hydrologic fields than subgrid-scale physics scheme choice. Results reveal that the WRF subgrid-scale physics configuration leads to larger spatiotemporal variance in simulated hydrometeorological conditions, whereas variance across meteorological forcing with common subgrid-scale physics configurations is more spatiotemporally constrained. Despite reasonably simulating precipitation, a delay in simulated discharge peak is due to a systematic cold bias across WRF simulations, suggesting the need for bias correction. Discharge shows greater variance in response to the WRF simulations across subgrid-scale physics schemes (26 %) rather than meteorological forcing (6 %). The topographic radiation option has minor effects on the watershed-average hydrometeorological processes but adds profound spatial heterogeneity to local energy budgets (±30 W m−2 in shortwave radiation and 1 K air temperature differences in late summer). This is the first presentation of sensitivity analyses that provide support to help guide the scientific community to develop observational constraints on atmosphere-through-bedrock processes and their interactions.

54 ENVIRONMENTAL SCIENCES↗

Distributed simulation, no special tools required

A tool kit of C language functions that can be linked with SIMSCRIPT programs to provide the data communication primitives necessary for distributed simulation is presented. A test case is discussed and some timing data are presented. Additionally some metrics, developed to determine the applicability of the server model decomposition for particular simulations, are discussed.

Paterra, Frank↗

Knowledge engineering for temporal dependency networks as operations procedures

This paper presents a case study of the knowledge engineering process employed to support the Link Monitor and Control Operator Assistant (LMCOA). The LMCOA is a prototype system which automates the configuration, calibration, test, and operation (referred to as precalibration) of the communications, data processing, metric data, antenna, and other equipment used to support space-ground communications with deep space spacecraft in NASA's Deep Space Network (DSN). The primary knowledge base in the LMCOA is the Temporal Dependency Network (TDN), a directed graph which provides a procedural representation of the precalibration operation. The TDN incorporates precedence, temporal, and state constraints and uses several supporting knowledge bases and data bases. The paper provides a brief background on the DSN, and describes the evolution of the TDN and supporting knowledge bases, the process used for knowledge engineering, and an analysis of the successes and problems of the knowledge engineering effort.

Fayyad, Kristina E.↗

Evaluation of Candidate Millimeter Wave Sensors for Synthetic Vision

The goal of the Synthetic Vision Technology Demonstration Program was to demonstrate and document the capabilities of current technologies to achieve safe aircraft landing, take off, and ground operation in very low visibility conditions. Two of the major thrusts of the program were (1) sensor evaluation in measured weather conditions on a tower overlooking an unused airfield and (2) flight testing of sensor and pilot performance via a prototype system. The presentation first briefly addresses the overall technology thrusts and goals of the program and provides a summary of MMW sensor tower-test and flight-test data collection efforts. Data analysis and calibration procedures for both the tower tests and flight tests are presented. The remainder of the presentation addresses the MMW sensor flight-test evaluation results, including the processing approach for determination of various performance metrics (e.g., contrast, sharpness, and variability). The variation of the very important contrast metric in adverse weather conditions is described. Design trade-off considerations for Synthetic Vision MMW sensors are presented.

Alexander, Neal T.↗