Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗
DOE OSTI2020

Exploring Hilbert space on a budget: Novel benchmark set and performance metric for testing electronic structure methods in the regime of strong correlation

This work explores the ability of classical electronic structure methods to efficiently represent (compress) the information content of full configuration interaction (FCI) wave functions. We introduce a benchmark set of four hydrogen model systems of different dimensionalities and distinctive electronic structures: a 1D chain, a 1D ring, a 2D triangular lattice, and a 3D close-packed pyramid. To assess the ability of a computational method to produce accurate and compact wave functions, we introduce the accuracy volume, a metric that measures the number of variational parameters necessary to achieve a target energy error. Using this metric and the hydrogen models, we examine the performance of three classical deterministic methods: (i) selected configuration interaction (sCI) realized both via an a posteriori (ap-sCI) and variational selection of the most important determinants, (ii) an a posteriori singular value decomposition (SVD) of the FCI tensor (SVD-FCI), and (iii) the matrix product state representation obtained via the density matrix renormalization group (DMRG). We find that the DMRG generally gives the most efficient wave function representation for all systems, particularly in the 1D chain with a localized basis. For the 2D and 3D systems, all methods (except DMRG) perform best with a delocalized basis, and the efficiency of sCI and SVD-FCI is closer to that of DMRG. For larger analogs of the models, the DMRG consistently requires the fewest parameters but still scales exponentially in 2D and 3D systems, and the performance of SVD-FCI is essentially equivalent to that of ap-sCI.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Blockchain based Communication Architectures with Applications to Private Security Networks

Existing communication protocols in high consequence security networks are highly centralized. While this naively makes the controls easier to physically secure, external actors require fewer resources to disrupt the system because there are fewer points in the system can be destroyed or interrupted without the entire system failing. We present a solution to this problem using a proof-of-work-based blockchain implementation built on MultiChain. We construct a test-bed network containing two types of data input: visual imagers and microwave sensor information. These data types are ubiquitous in perimeter intrusion detection security systems and allow a realistic representation of a real-world network architecture. The cameras in this system use an object detection algorithm to nd important targets in the scene. The raw data from the camera and the outputs from the detection algorithm are then placed in a transaction on the distributed ledger. Similarly, microwave data is used to detect relevant events and are placed in a transaction. These transactions are then bundled into blocks and broadcast to the rest of the network using the Bitcoin-based MultiChain protocol. We develop five tests to examine the security metrics of our network. We performed the five security metric test using different sized networks from 7 to 39 nodes to determine how the metrics scale with respect to size. We nd that when compared to a centralized architecture our implementation provides a resiliency increase that is expected from a blockchain-based protocol without slowing the system so much that a human operator would notice. Furthermore, our approach is able to detect tampering in real time. Based on these results, we theorize that security networks in general could use a blockchain-based approach in a meaningful way.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Development of an integrated platform for hardware-in-the-loop evaluation of microgrids prior to site commissioning

This paper presents an integrated hardware-in-the-loop (HIL) platform for testing the operation and control of a real-world microgrid system prior to site commissioning. The proposed testing approach shows the value of setting up an integrated HIL platform to test multiple hardware devices (including the system-level controller, device controllers, and field devices). A test bed developed for the Borrego Springs community microgrid is used as an example to demonstrate the feasibility of the integrated platform. Comprehensive tests are carried out using the Borrego Springs test bed to validate the platform, meet the test objectives, and gain valuable insight prior to site commissioning. The selected test scenarios, initial conditions, events, test metrics, etc., are similar to those defined in IEEE P2030.8. The test objectives are to evaluate the microgrid controller and the operation of the microgrid. The HIL platform presented in this paper shows good value for microgrid testing in terms of its flexibility and scalability for testing various microgrids; its accuracy in its representation of the field; the comprehensiveness of the testing capability; and the efficiency of testing, which can reduce risks prior to field deployment.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Towards Scaling Law Analysis For Spatiotemporal Weather Data

Compute-optimal scaling laws are relatively well studied for NLP and CV, where objectives are typically single-step and targets are comparatively homogeneous. Weather forecasting is harder to characterize in the same framework: autoregressive rollouts compound errors over long horizons, outputs couple many physical channels with disparate scales and predictability, and globally pooled test metrics can disagree sharply with per-channel, late-lead behavior implied by short-horizon training. We extend neural scaling analysis for autoregressive weather forecasting from single-step training loss to long rollouts and per-channel metrics. We quantify (1) how prediction error is distributed across channels and how its growth rate evolves with forecast horizon, (2) if power law scaling holds for test error, relative to rollout length when error is pooled globally, and (3) how that fit varies jointly with horizon and channel for parameter, data, and compute-based scaling axes. We find strong cross-channel and cross-horizon heterogeneity: pooled scaling can look favorable while many channels degrade at late leads. We discuss implications for weighted objectives, horizon-aware curricula, and resource allocation across outputs.

Kiefer Jr, Alexander [ORNL] (ORCID:000000025398874↗
DOE OSTI2025

Recrystallization driven softening and heating rate dependencies of FeCrAl nuclear fuel cladding during accident transients

A refined understanding of FeCrAl cladding behavior during rapid transients is critical for its potential deployment in light-water reactors. Current assessments focus on transient burst testing metrics such as balloon geometry, burst temperature, and hoop stress, often used as proxies for simpler conventional tensile properties. However, directly correlating isothermal tensile and creep data with accident transient scenarios remains a challenge, although it is essential for high fidelity model development. Recent modeling based on tensile tests up to 800 °C, conducted with both immediate loading and a 10-minute soak, showed that immediate loading better predicts experimental burst temperatures, indicating a thermal softening effect. Building upon this observation, the current study connects transient performance, microstructural evolution, and high-temperature tensile properties by leveraging results from C26M claddings burst tests performed at heating rates of 1–50 °C/s and hoop stresses from 25 to 100 MPa. At 25 MPa, rupture temperatures varied by only 6 °C, but at 100 MPa, the difference reached 116 °C, with faster heating yielding higher burst temperatures. Microstructural analysis identified recrystallization as the primary cause of heating rate-dependent softening, eliminating prior cold-working. In-situ thermomechanical data linked ballooning onset to localized instabilities, similar to ultimate tensile strength behavior in conventional tensile tests. High heating rates correlated with immediate loading tensile data, while lower rates matched soaked data. Furthermore, by linking burst performance to microstructural evolution and tensile properties, this work provides a foundation for more accurate modeling of FeCrAl claddings and potentially other Fe-based materials under accident conditions.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2022

Detecting impacts of surface development near weather stations since 1895 in the San Joaquin Valley of California

Abstract Temperature readings observed at surface weather stations have been used for detecting changes in climate due to their long period of observations. The most common temperature metrics recorded are the daily maximum (TMax) and minimum (TMin) extremes. Unfortunately, influences besides background climate variations impact these measurements such as changes in (1) instruments, (2) location, (3) time of observation, and (4) the surrounding artifacts of human civilization (buildings, farms, streets, etc.) Quantifying (4) is difficult because the surrounding infrastructure, unique to each site, often changes slowly and variably and is thus resistant to general algorithms for adjustment. We explore a direct method of detecting this impact by comparing a single station that experienced significant development from 1895 to 2019, and especially since 1970, relative to several other stations with lesser degrees of such development (after adjustments for the (1) to (3) are applied). The target station is Fresno, California (metro population ~ 15,000 in 1900 and ~ 1 million in 2019) situated on the eastern side of the broad, flat San Joaquin Valley in which several other stations reside. A unique component of this study is the use of pentad (5-day averages) as the test metric. Results indicate that Fresno experienced + 0.4 °C decade −1 more nighttime warming (TMin) since 1970 than its neighbors—a time when population grew almost 300%. There was little difference seen in TMax trends between Fresno and non-Fresno stations since 1895 with TMax trends being near zero. A case is made for the use of TMax as the preferred climate metric relative to TMin for a variety of physical reasons. Additionally, temperatures measured at systematic times of the day (i.e., hourly) show promise as climate indicators as compared with TMax and especially TMin (and thus TAvg) due to several complicating factors involved with daily high and low measurements.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

A comparison of surrogate constitutive models for viscoplastic creep simulation of HT-9 steel

Mechanistic microstructure-informed constitutive models for the mechanical response of polycrystals are a cornerstone of computational materials science. However, as these models become increasingly more complex – often involving coupled differential equations describing the effect of specific deformation modes – their associated computational costs can become prohibitive, particularly in optimization or uncertainty quantification tasks that require numerous model evaluations. To address this challenge, surrogate constitutive models that balance accuracy and computational efficiency are highly desirable. Data-driven surrogate models, that learn the constitutive relation directly from data, have emerged as a promising solution. In this work, we develop two local surrogate models for the viscoplastic response of a steel: a piecewise response surface method and a mixture of experts model. These surrogates are designed to adapt to complex material behavior, which may vary with material parameters or operating conditions. The surrogate constitutive models are applied to creep simulations of HT-9 steel, an alloy of considerable interest to the nuclear energy sector due to its high tolerance to radiation damage, using training data generated from viscoplastic self-consistent (VPSC) simulations. In conclusion, we define a set of test metrics to numerically assess the accuracy of our surrogate models for predicting viscoplastic material behavior, and show that the mixture of experts model outperforms the piecewise response surface method in terms of accuracy.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Neural network-based classification and regression of magnetohydrodynamic modes in tokamaks

We present a machine learning-based magnetohydrodynamic (MHD) classifier and regressor that utilizes real or complex-valued 3D magnetic sensor array data to determine neoclassical tearing mode (NTM) onset times in tokamaks with millisecond accuracy. The input dataset consists of poloidal profiles of complex Fourier amplitudes with an n = 1 toroidal mode number from 144 human-labeled ITER Baseline Scenario discharges in the DIII-D tokamak, spanning both tearing-dominated and sawtooth-dominated regimes. Since m, n = 2,1 NTMs frequently emerge alongside sawteeth at the same frequency in this scenario, the focus is on isolating the m = 1 and m = 2 components of the n = 1 MHD mode near the tearing onset. To improve model regularization and prediction stability, singular value decomposition was applied to balance the sawtooth and tearing datasets. The enriched datasets facilitated training neural networks that learn the key distinguishing features of sawtooth and tearing modes in the poloidal profiles of their magnetic amplitude and phase. When the modes occur independently, the networks achieve perfect classification due to the modes’ distinct characteristics and low measurement noise. In the more experimentally relevant case where both modes coexist, the networks maintain exceptional performance across key metrics. Tests on synthetic data with known ground truth demonstrate the superior accuracy of the neural network trained on complex-valued input compared to models using real amplitude, phase, or pseudo-complex data, achieving both a mean time delay and standard deviation below 1 ms. Notably, standard linear regression methods fitting the dominant singular modes to the data closely match the neural network’s performance. Applying these methods across a broad range of H-mode scenarios will enable future studies to systematically identify dominant NTM triggers as scenario-specific variables, paving the way for more effective tearing mode avoidance strategies in future fusion reactor designs.

machine learning↗
DOE OSTI2023

Trust Model System for the Energy Grid of Things Network Communications

Network communication is crucial in the Energy Grid of Things (EGoT). Without a network connection, the energy grid becomes just a power grid where the energy resources are available to the customer uni-directionally. A mechanism to analyze and optimize the energy usage of the grid can only happen through a medium, a communications network, that enables information exchange between the grid participants and the service provider. Security implementers of EGoT network communication take extraordinary measures to ensure the safety of the energy grid, a critical infrastructure, as well as the safety and privacy of the grid participants. With the dynamic nature of network communication of the EGoT, the information provided by the customer or the service provider can be falsified by a malicious attacker. Therefore, a trust model is necessary to monitor any abnormal activities. This paper describes a distributed trust model system that meets the need of the EGoT. This paper describes methods for evaluating and improving the distributed trust model using standard hypothesis testing metrics such as true positive, false positive, true negative, false negative, equal error rate, and F1 score. Example calculations are shown based on generated sample data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Data-Driven Protection Software to classify fault locations by protective zone in distribution systems with high PV penetration

The software contains (a) the source codes to generate Point-on-Wave (PoW) transient data for any feeder model in Alternative Transient Program (ATP) format. Codes provide options to change different steady state settings, including the loading condition and PV capacity and transient state setting like faults type, location and initiation time (b) data post-processing source code to converted data from native format to COMTRADE, csv, HDF5 (c) Docker container to train CNN to classify fault locations by protective zone. The container takes dataset and other training parameters (sampling rate, training epochs, batch size etc) as input to train CNN. The container writes back the trained CNN model, training and testing metrics and plots to the local workstation

Ramesh, Meghana↗
DOE OSTI2021

A hierarchical evaluation framework for assessing climate simulations relevant to the energy-water-land nexus (Final Report)

The overarching goal was to construct a hierarchy of new and well-tested metrics and analysis tools that support both fundamental and use-inspired research. Motivating this goal was a convergence of needs from climate scientists and stakeholders alike for a systematic, robust framework of model evaluation and diagnosis to provide scientific insights, inform model development, support best practices for the use of climate model outputs, and facilitate communication of climate information in the evolving landscapes of multi-model, multi-resolution, and large ensemble simulations that generate terabytes of data for any single climate run. Steps toward attaining these goals benefited from expertise and established capability of the project team, which included leadership of the North American Regional Climate Change Assessment Program (NARCCAP) and the Coordinated Regional Downscaling Experiment (CORDEX), development of hierarchical model evaluation approaches, and successful research in the analysis and diagnosis of climate model skill, as well as the understanding and modeling of regional climate processes in North America. As part of the overarching goal, the project worked to disseminate a suite of methodologies, algorithms, and software components that the wider community can employ to advance climate science and applications. With rigorous demonstration, the evaluation framework and the mix of standard and high risk / high reward approaches helped form the basis for future development of a computationally enabled user-friendly system for community use.

17 WIND ENERGY↗
DOE OSTI2025

REDESIGNING A PERFORMANCE MONITORING SOFTWARE FOR SUPERCOMPUTERS

The objective of this project was to improve upon the existing Watchr software that charts performance test metrics from the Trilinos project run on supercomputers at Sandia and elsewhere across the DOE complex. Software was iteratively designed and developed using Python Pandas and Dash data visualization to improve the extensibility and user experience of Watchr. Documentation is being maintained for future developers who want to extend the application.

Camacho, Dane Joseph [Sandia National Laboratories↗
DOE OSTI2023

Ultimate FY23 Vacuum Creep Testing Facility Operation Report

This facility operation report covers the refurbishment and initial testing of a vacuum creep frame used to conduct 1300°C tensile and creep tests on Mo- and Nb-based alloys. The report details the refurbishment of the frame and initial operation including 100 h creep experiments compared to similar tests in argon. Unlike the 25 mm dogbone specimens tested in argon, the vacuum load train requires 76 mm dog bone specimens. Testing in vacuum has eliminated concern of O embrittlement in argon and Nb alloy CB103 showed >120% tensile elongation at 1300°C and 1450°C. Also, a window in the vacuum chamber allows the use of digital image correlation (DIC) to measure strain rather than the load train displacement used in the argon frames. Comparison of the displacement and DIC results in the vacuum frame showed significant variation in primary creep deformation, likely due to the machine compliance, but similar secondary creep rates. Recently, ULTIMATE teams have submitted specimens for testing in vacuum. With the relatively low volume of creep testing, the facility performance metric of testing within 4 weeks of receiving specimens has been met.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Review of Wave Energy Converter Power Take-Off Systems, Testing Practices, and Evaluation Metrics: Preprint

While the field of wave energy has been the subject of numerical simulation, scale model testing, and precommercial project testing for decades, wave energy technologies remain in the early stages of development and must continuing proving themselves as a promising modern renewable energy field. One of the difficulties that wave energy systems have been struggling to overcome is the design of highly efficient energy conversion systems that can convert the mechanical power, derived from the oscillation of wave activated bodies, into another useful product. Often the power take-off (PTO) is defined as the single unit responsible for converting mechanical power into another usable form such as electricity, pressurized fluid, compressed air, and others. The PTO, and the entire power conversion chain (PCC), is of great importance as it affects not only how efficient wave power is converted into electricity, but also contributes to the mass, size, structural dynamics, and levelized cost of energy (LCOE) of the wave energy converter (WEC). Unlike wind and solar, there is no industrial standard device, or devices, for wave energy conversion and this diversity is transferred to the PTO system. The majority of current WEC PTO systems incorporate a mechanical or hydraulic drive train, power generator, and an electrical control system. The challenge of WEC PTO designs is designing a mechanical-to-electrical component that can efficiently convert irregular, bi-directional, low frequency and low alternating velocity wave motions. While gross average power levels can be predicted in advance, the variable wave elevation input has to be converted into smooth electrical output and hence usually necessitates some type of energy storage system, such as battery storage, accumulator super capacitors, etc., or other means of compensation such as an array of devices. One of the primary challenges for wave energy converter systems is the fluctuating nature of wave resources, which require WEC components to be designed to handle loads (i.e. torques, forces, and powers) that are many times greater than the average load. This approach requires a much greater PTO capacity than the average power output and indicates a higher cost. In addition, supporting mechanical coupling and or gearing can be added to the PCC to help alleviate the difficulties with transmission and control of fluctuating large loads with low frequencies (indicative of wave forcing) into smaller loads at higher frequencies (optimum for conventional electrical machine design) can quickly increase the complexity of the PCC which could result in a greater number of failure modes and increased maintenance costs. All of the previous points demonstrate how the PTO influences WEC dynamics, reliability, performance and cost which are critical design factors. This paper further explores these topics by providing a review of the state-of-the-art PTO systems currently under development, how these novel PTO systems are tested and derisked prior to precommercial deployment, and the evaluation metrics historically used to differentiate between PTO designs and how they can be improved to support control co-design focused development of wave energy systems.

laboratory testing↗
DOE OSTI2022

Scalability Testing Approach for Internet of Things for Manufacturing SQL and NoSQL Database Latency and Throughput

The proliferation of low-cost sensors and industrial data solutions has continued to push the frontier of manufacturing technology. Machine learning and other advanced statistical techniques stand to provide tremendous advantages in production capabilities, optimization, monitoring, and efficiency. The tremendous volume of data gathered continues to grow, and the methods for storing the data are critical underpinnings for advancing manufacturing technology. This work aims to investigate the ramifications and design tradeoffs within a decoupled architecture of two prominent database management systems (DBMS): sql and NoSQL. A representative comparison is carried out with Amazon Web Services (AWS) DynamoDB and AWS Aurora MySQL. The technologies and accompanying design constraints are investigated, and a side-by-side comparison is carried out through high-fidelity industrial data simulated load tests using metrics from a major US manufacturer. The results support the use of simulated client load testing for comparing the latency of database management systems as a system scales up from the prototype stage into production. As a result of complex query support, MySQL is favored for higher-order insights, while NoSQL can reduce system latency for known access patterns at the expense of integrated query flexibility. Here, by reviewing this work, a manufacturer can observe that the use of high-fidelity load testing can reveal tradeoffs in IoTfM write/ingestion performance in terms of latency that are not observable through prototype-scale testing of commercially available cloud DB solutions.

AWS↗
DOE OSTI2022

Improving energy performance metrics to maximize the benefits of disruptive technologies

Advances in appliance and equipment systems' technologies also cause changes in their technical characteristics that can affect energy consumption and associated greenhouse gas emissions. Energy performance evaluation methods that use testing and metrics can evolve to account for energy consumption contributed from new technological characteristics. However, there is no systematic effort to enable the advancement of test procedures and performance metrics to maximize the energy, climate, and financial benefits of disruptive technologies. Therefore, it is important for countries to regularly update their energy-efficiency policy programs, such as standards and labels, by improving test procedures and metrics to reflect the innovation in emerging technologies as well as to mitigate the risk of deploying obsolete technology. This paper shows how energy performance metrics have been improving in selected appliances and equipment and provides insights to design a systematic effort to improve metrics and test procedures at the speed of ongoing changes in technologies and markets.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

First Sagittarius A* Event Horizon Telescope Results. VI. Testing the Black Hole Metric

Astrophysical black holes are expected to be described by the Kerr metric. This is the only stationary, vacuum, axisymmetric metric, without electromagnetic charge, that satisfies Einstein’s equations and does not have pathologies outside of the event horizon. We present new constraints on potential deviations from the Kerr prediction based on 2017 EHT observations of Sagittarius A* (Sgr A*). We calibrate the relationship between the geometrically defined black hole shadow and the observed size of the ring-like images using a library that includes both Kerr and non-Kerr simulations. We use the exquisite prior constraints on the mass-to-distance ratio for Sgr A* to show that the observed image size is within ∼10% of the Kerr predictions. We use these bounds to constrain metrics that are parametrically different from Kerr, as well as the charges of several known spacetimes. To consider alternatives to the presence of an event horizon, we explore the possibility that Sgr A* is a compact object with a surface that either absorbs and thermally reemits incident radiation or partially reflects it. Using the observed image size and the broadband spectrum of Sgr A*, we conclude that a thermal surface can be ruled out and a fully reflective one is unlikely. We compare our results to the broader landscape of gravitational tests. Together with the bounds found for stellar-mass black holes and the M87 black hole, our observations provide further support that the external spacetimes of all black holes are described by the Kerr metric, independent of their mass.

79 ASTRONOMY AND ASTROPHYSICS↗