Search NASA⌕ Search

SEARCH · Search NASA

Results for “Metrics and Validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Validation Metrics for Fixed Effects and Mixed-Effects Calibration

The modern scientific process often involves the development of a predictive computational model. To improve its accuracy, a computational model can be calibrated to a set of experimental data. A variety of validation metrics can be used to quantify this process. Some of these metrics have direct physical interpretations and a history of use, while others, especially those for probabilistic data, are more difficult to interpret. In this work, a variety of validation metrics are used to quantify the accuracy of different calibration methods. Frequentist and Bayesian perspectives are used with both fixed effects and mixed-effects statistical models. Through a quantitative comparison of the resulting distributions, the most accurate calibration method can be selected. Two examples are included which compare the results of various validation metrics for different calibration methods. It is quantitatively shown that, in the presence of significant laboratory biases, a fixed effects calibration is significantly less accurate than a mixed-effects calibration. This is because the mixed-effects statistical model better characterizes the underlying parameter distributions than the fixed effects model. The results suggest that validation metrics can be used to select the most accurate calibration model for a particular empirical model with corresponding experimental data.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2018

Quantification of Dynamic Model Validation Metrics Using Uncertainty Propagation from Requirements

The Space Launch System, NASA's new large launch vehicle for long range space exploration, is presently in the final design and construction phases, with the first launch scheduled for 2019. A dynamic model of the system has been created and is critical for calculation of interface loads and natural frequencies and mode shapes for guidance, navigation, and control (GNC). Because of the program and schedule constraints, a single modal test of the SLS will be performed while bolted down to the Mobile Launch Pad just before the first launch. A Monte Carlo and optimization scheme will be performed to create thousands of possible models based on given dispersions in model properties and to determine which model best fits the natural frequencies and mode shapes from modal test. However, the question still remains as to whether this model is acceptable for the loads and GNC requirements. An uncertainty propagation and quantification (UP and UQ) technique to develop a quantitative set of validation metrics that is based on the flight requirements has therefore been developed and is discussed in this paper. There has been considerable research on UQ and UP and validation in the literature, but very little on propagating the uncertainties from requirements, so most validation metrics are "rules-of-thumb;" this research seeks to come up with more reason-based metrics. One of the main assumptions used to achieve this task is that the uncertainty in the modeling of the fixed boundary condition is accurate, so therefore that same uncertainty can be used in propagating the fixed-test configuration to the free-free actual configuration. The second main technique applied here is the usage of the limit-state formulation to quantify the final probabilistic parameters and to compare them with the requirements. These techniques are explored with a simple lumped spring-mass system and a simplified SLS model. When completed, it is anticipated that this requirements-based validation metric will provide a quantified confidence and probability of success for the final SLS dynamics model, which will be critical for a successful launch program, and can be applied in the many other industries where an accurate dynamic model is required.

Brown, Andrew M.↗
DOE OSTI2024

Validation Assessment of Turbulent Reacting Flow Model Using The Area-Validation Metric on Medium-Scale Methanol Pool Fire Results

Accident analysis and ensuring power plant safety are pivotal in the nuclear energy sector. Significant strides have been achieved over the past few decades regarding fire protection and safety, primarily centered on design and regulatory compliance. Yet, after the Fukushima accident a decade ago, the imperative to enhance measures against fire, internal flooding, and power loss has intensified. Hence, a comprehensive, multilayered protection strategy against severe accidents is needed. Consequently, gaining a deeper insight into pool fires and their behavior through extensive validated data can greatly aid in improving these measures using advanced validation techniques. A model validation study was performed at Sandia National Laboratories in which a 30-cm diameter methanol pool fire was modeled using the SIERRA/Fuego turbulent reacting flow code. This validation study used a standard validation experiment to compare model results against, and conclusions have been published. The fire was modeled with a Large Eddy Simulation (LES) turbulence model with subgrid turbulent kinetic energy closure. Combustion was modeled using a strained laminar flamelet library approach. Radiative heat transfer was accounted for with a model utilizing the gray-gas approximation. In the present study, additional validation analysis is performed using the area validation metric (AVM). These activities are done on multiple datasets involving different variables and temporal/spatial ranges and intervals. In conclusion, the results provide insight into the use of the area validation metric on such temporally varying datasets and the importance of physics-aware use of the metric for proper analysis.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
NASA NTRS2010

ISS Logistics Hardware Disposition and Metrics Validation

I was assigned to the Logistics Division of the International Space Station (ISS)/Spacecraft Processing Directorate. The Division consists of eight NASA engineers and specialists that oversee the logistics portion of the Checkout, Assembly, and Payload Processing Services (CAPPS) contract. Boeing, their sub-contractors and the Boeing Prime contract out of Johnson Space Center, provide the Integrated Logistics Support for the ISS activities at Kennedy Space Center. Essentially they ensure that spares are available to support flight hardware processing and the associated ground support equipment (GSE). Boeing maintains a Depot for electrical, mechanical and structural modifications and/or repair capability as required. My assigned task was to learn project management techniques utilized by NASA and its' contractors to provide an efficient and effective logistics support infrastructure to the ISS program. Within the Space Station Processing Facility (SSPF) I was exposed to Logistics support components, such as, the NASA Spacecraft Services Depot (NSSD) capabilities, Mission Processing tools, techniques and Warehouse support issues, required for integrating Space Station elements at the Kennedy Space Center. I also supported the identification of near-term ISS Hardware and Ground Support Equipment (GSE) candidates for excessing/disposition prior to October 2010; and the validation of several Logistics Metrics used by the contractor to measure logistics support effectiveness.

Rogers, Toneka R.↗
DOE OSTI2021

Cumulative χ 2 Metric for VALID for ENDF/B-VII.1 and ENDF/B-VIII.0 in SCALE 6.3b9 [Slides]

Quantifying the performance of a new evaluation is difficult and doing so for a new library is significantly more challenging. Cumulative χ 2 metric is an established approach to comparing distributions. In this case, C/E results from different libraries. Andrei Trkov and Roberto Capote have published cumulative χ 2 results for ENDF/B-VIII.0. Cumulative χ 2 metric was recently calculated for VALID library maintained at ORNL.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

ENDF/B-VIII.1 Validation (Preliminary Release 2 + New 239 Pu)

This document should serve as a reference for validation testing of nuclear data files included in the second preliminary release of ENDF/B-VIII.1 and the newly evaluated 239 Pu nuclear data file to be included in the third preliminary release of ENDF/B-VIII.1. Experiment names listed in this report are based on International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook designations. Additional information on ICSBEP handbook designations can be found at https://www.oecd-nea.org/jcms/pl_20291/icsbep-handbook. The validation metrics used in this report include the effective neutron multiplication factor, k eff , and tritium production. All validation metrics were calculated using the radiation transport code Monte Carlo N-Particle (MCNP).

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2026

Assessing Metal Ion Assignment Accuracy in Protein Data Bank Models via Elemental Spectroscopy

Accurate representation of metal ions in macromolecular structures is critical for chemical interpretation, computational modeling, and machine-learning methods that rely on Protein Data Bank (PDB) entries. However, the elemental identity of metals modeled in crystallographic structures is often inferred indirectly and rarely validated experimentally. Here, we combine Particle Induced X-ray Emission (PIXE) and X-ray Fluorescence Spectroscopy (XRFS) to determine the elemental composition of protein samples used to generate 70 deposited metalloprotein crystal structures. By analyzing the original protein material employed for crystallization, but before the addition of crystallization buffer solutions, we assess whether the modeled metal ions in deposited structures are consistent with experimentally detectable elemental content. We find that in a majority of cases, the metals modeled in the corresponding PDB entries are inconsistent with the metals present in the protein samples before crystallization, or that additional metals are present but not represented in the structural models. Spectroscopic results were integrated with automated crystallographic validation metrics, including real-space Z-difference (RSZD) analysis and systematic rerefinement, to evaluate atomic-number mismatch at metal sites. PIXE and XRFS show strong agreement for dominant elemental signals and provide complementary, scalable approaches for identifying suspect metal assignments. This work does not address physiological or functional metalation but instead highlights a widespread data integrity issue in deposited macromolecular structures, PDB-wide. These results establish an experimentally corroborated link between elemental identity and crystallographic validation metrics, enabling the large-scale detection of chemically inconsistent annotations in structural databases used for computational modeling and machine learning.

Crystallization↗
DOE OSTI2021

CERES : a cryo-EM re-refinement system for continuous improvement of deposited models

The field of electron cryomicroscopy (cryo-EM) has advanced quickly in recent years as the result of numerous technological and methodological developments. This has led to an increase in the number of atomic structures determined using this method. Recently, several tools for the analysis of cryo-EM data and models have been developed within the Phenix software package, such as phenix.real_space_refine for the refinement of atomic models against real-space maps. Also, new validation metrics have been developed for low-resolution cryo-EM models. To understand the quality of deposited cryo-EM structures and how they might be improved, models deposited in the Protein Data Bank that have map resolutions of better than 5 Å were automatically re-refined using current versions of Phenix tools. The results are available on a publicly accessible web page (https://cci.lbl.gov/ceres). The implementation of a Cryo-EM Re-refinement System ( CERES ) for the improvement of models deposited in the wwPDB, and the results of the re-refinements, are described. Based on these results, contents are proposed for a `cryo-EM Table 1', which summarizes experimental details and validation metrics in a similar way to `Table 1' in crystallography. The consistent use of robust metrics for the evaluation of cryo-EM models and data should accompany every structure deposition and be reported in scientific publications.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Cumulative $\chi^2$ Metric for VALID for ENDF/B-VII.1 and ENDF/B-VIII.0 in SCALE 6.3b9

Nuclear data testing is generally performed in a manner similar to validation: a series of critical benchmark experiments is modeled, and the differences between calculated and expected results are examined to understand the performance of a code and cross section set. This process is repeated for multiple sets of experiments with different transport codes and sometimes with different datasets. Occasionally, data processing differences are also important in understanding the differences in calculated results. Relevant experts then meet periodically to compare results and set priorities for improvement in the next round of data updates. This process has been largely effective, despite the confounding factors of different benchmark sets, application areas, sponsors, nationalities, and the resulting different priorities.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS2022

Combining Machine Learning and Numerical Simulation for High-Resolution PM2.5 Concentration Forecast

Forecasting ambient PM2.5 concentrations with spatiotemporal coverage is key to alerting decision-makers of pollution episodes and preventing detrimental public exposure, especially in regions with limited ground air monitoring stations. The existing methods either rely on chemical transport models (CTMs) to forecast spatial distribution of PM2.5 with nontrivial uncertainty or statistical algorithms to forecast PM2.5 concentration time-series at air monitoring locations without continuous spatial coverage. In this study, we developed a PM2.5 forecast framework by combining the robust Random Forest algorithm with a publicly accessible global CTM forecast product – NASA’s Goddard Earth Observing System “Composition Forecasting” (GEOS-CF), providing spatiotemporally continuous PM2.5 concentration forecasts for the next five days at a 1-km spatial resolution. Our forecast experiment was conducted for a region in Central China including the populous and polluted Fenwei Plain. The forecast for the next two days had overall validation R2 of 0.76 and 0.64, respectively; the R2 was around 0.5 for the following three forecast days. Spatial cross-validation showed similar validation metrics. Our forecast model, with validation normalized mean bias close to zero, substantially reduced the large biases in GEOS-CF. The proposed framework requires minimal computational resources compared to running CTMs at urban scales, enabling near-real-time PM2.5 forecast in resource-restricted environments.

PM2.5↗
NASA NTRS2009

Disruption Tolerant Networking Flight Validation Experiment on NASA's EPOXI Mission

In October and November of 2008, the Jet Propulsion Laboratory installed and tested essential elements of Delay/Disruption Tolerant Networking (DTN) technology on the Deep Impact spacecraft. This experiment, called Deep Impact Network Experiment (DINET), was performed in close cooperation with the EPOXI project which has responsibility for the spacecraft. During DINET some 300 images were transmitted from the JPL nodes to the spacecraft. Then they were automatically forwarded from the spacecraft back to the JPL nodes, exercising DTN's bundle origination, transmission, acquisition, dynamic route computation, congestion control, prioritization, custody transfer, and automatic retransmission procedures, both on the spacecraft and on the ground, over a period of 27 days. All transmitted bundles were successfully received, without corruption. The DINET experiment demonstrated DTN readiness for operational use in space missions. This activity was part of a larger NASA space DTN development program to mature DTN to flight readiness for a wide variety of mission types by the end of 2011. This paper describes the DTN protocols, the flight demo implementation, validation metrics which were created for the experiment, and validation results.

DTN↗
DOE OSTI2023

Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets

Abstract With the increasingly more important role of machine learning (ML) models in chemical research, the need for putting a level of confidence to the model predictions naturally arises. Several methods for obtaining uncertainty estimates have been proposed in recent years but consensus on the evaluation of these have yet to be established and different studies on uncertainties generally uses different metrics to evaluate them. We compare three of the most popular validation metrics (Spearman’s rank correlation coefficient, the negative log likelihood (NLL) and the miscalibration area) to the error-based calibration introduced by Levi et al. ( Sensors 2022 , 22 , 5540). Importantly, metrics such as the negative log likelihood (NLL) and Spearman’s rank correlation coefficient bear little information in themselves. We therefore introduce reference values obtained through errors simulated directly from the uncertainty distribution. The different metrics target different properties and we show how to interpret them, but we generally find the best overall validation to be done based on the error-based calibration plot introduced by Levi et al. Finally, we illustrate the sensitivity of ranking-based methods (e.g. Spearman’s rank correlation coefficient) towards test set design by using the same toy model ferent test sets and obtaining vastly different metrics (0.05 vs. 0.65).

Rasmussen, Maria H.↗
NASA NTRS2020

An Introduction to the NASA GMAO Coupled Atmosphere-Ocean System - GEOS-S2S Version 3

Recently NASA's Global Modeling and Assimilation Office (GMAO) has developed a new Subseasonal to Seasonal Prediction system Version 3 (GEOS-S2S-3). This upgrade replaces the GEOS-S2S-2 which is NASA's current contribution to the North American Multi-Model Experiment seasonal prediction project (Kirtman et al., 2014). The main improvements for our S2S-3 system include 1) a higher resolution MOM5 (Griffies et al., 2005) ocean model (now 0.25o x 0.25o x 50 layers), 2) an improved atmospheric/ocean interface layer (Akella and Suarez, 2018), and 3) assimilation of a long-track satellite salinity into the ocean model (Hackert et al, 2019). Atmospheric forcing is provided by the NASA MERRA-2 reanalysis (Gelaro et al., 2017). Initialization for the ocean relies on the GMAO ocean reanalysis system which assimilates all available in situ temperature and salinity, satellite sea surface salinity, and sea level using the Local Ensemble Transform Kalman Filter (LETKF) implementation of (Penny et al., 2013) on a 5 day assimilation cycle with 20 fixed ensemble members.In this presentation, we will authenticate our new S2S-3 ocean reanalysis using standard GODAE validation metrics. For example, we will compare gridded fields of mean and standard deviation of the ocean reanalysis versus observed fields. We will show correlation/RMS of model versus observations and temperature and salinity mean profiles for the various basins and latitude bands. Basin-scale volume transports, such as the Atlantic Meridional Overturning Circulation and the Indonesian Throughflow will be validated. Equatorial ocean waves will be compared by decomposing sea level into Kelvin and Rossby components. For each of these metrics, we plan to validate the results and then compare our new S2S-3 against the current production version, S2S-2. Finally, we will compare 9-month seasonal forecasts initialized from these two systems for the tropical Pacific NINO3.4 region over the period 1981-present.

Molod, Andrea↗
DOE OSTI2023

A Data Processing Pipeline for Adversarial Socio-Technical Network Analysis

With the rapid adoption of emerging technologies, there is a need to catalog and model sociotechnical interdependencies that have been historically used to influence the operation of Critical Infrastructure networks including the impacts of mergers and acquisitions, hostile takeovers, and foreign investment. Our research intends to address this need with two primary contributions. First, we have developed a data curation and processing pipeline to generate sociotechnical networks extracted from a variety of data sources including SEC filings and infrastructure asset databases. The pipeline, implemented in Apache Airflow, extracts and normalizes the representation of entities and relations, specified within ontologies. Our intent is to provide an extensible, machine-actionable approach to quickly communicate such models, reproduce previous results, and adapt them to new, unanticipated situations. Second, networks produced by our pipeline enable the development of graph-theoretic metrics that consider the properties of network components in addition to its topology. Metadata associated with network components---whether semantic, temporal, or geospatial---affects the alignment of generated networks with assumptions underlying complexity metrics. Validation of generated networks relative to component types defined by an ontology, may allow the research community to adapt metrics to the semantics of the domains being studied. Generated networks may be processed as knowledge, dynamic, or spatial graphs and enables a variety of analyses including automated reasoning and measures of network complexity. Automated reasoning views extracted entities and relations as a knowledge graph; this enables application of inference rules that represent historically-attested adversarial business methods and applies that behavior to a specific geographic context. Measures of network complexity, including degree distribution, reachability analyses, temporal analysis, and community detection can be adapted to indicate adversarial organizational influence.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Templates for Risk Informed Assurance with Curvature Embeddings (TRACE)

We investigate recovery of geometric structure from networks embedded in manifolds with spatially varying curvature, extending the constant-curvature framework of Lubold et al. (2023). Our work supports cascade risk assessment in critical infrastructure through the Templates for Risk-informed Assurance with Curvature Embeddings (TRACE) framework. Simulations on a bi-modal Gaussian surface show that constant-curvature methods yield weighted averages shaped by clique patterns, while hierarchical clustering identifies distinct regimes. Localized estimation, however, reveals boundary contamination in transitional regions. To address heterogeneity, we develop distance metrics for graphs with edge and node features, proving their metric validity, and validate them via deterministic graph generation from canonical tilings. We further propose a diffusion-based anomaly detection approach that treats networks as glued manifolds, using curvature discontinuities to detect structural anomalies. Employing the carré-du-champ operator and scalar curvature, we achieve robust anomaly discrimination, demonstrated on the Singapore Water Treatment (SWaT) dataset with joint network-traffic and sensor features. Integration with TRACE reveals how curvature shapes cascade dynamics: positive curvature impedes, while negative curvature accelerates propagation. This geometric perspective provides interpretable risk metrics and visualization tools for critical infrastructure managers. While full validation remains ongoing, our contributions establish a rigorous foundation for geometric analysis of network resilience and cascade vulnerability.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2011

Validating LES for Jet Aeroacoustics

Engineers charged with making jet aircraft quieter have long dreamed of being able to see exactly how turbulent eddies produce sound and this dream is now coming true with the advent of large eddy simulation (LES). Two obvious challenges remain: validating the LES codes at the resolution required to see the fluid-acoustic coupling, and the interpretation of the massive datasets that are produced. This paper addresses the former, the use of advanced experimental techniques such as particle image velocimetry (PIV) and Raman and Rayleigh scattering, to validate the computer codes and procedures used to create LES solutions. This paper argues that the issue of accuracy of the experimental measurements be addressed by cross-facility and cross-disciplinary examination of modern datasets along with increased reporting of internal quality checks in PIV analysis. Further, it argues that the appropriate validation metrics for aeroacoustic applications are increasingly complicated statistics that have been shown in aeroacoustic theory to be critical to flow-generated sound, such as two-point space-time velocity correlations. A brief review of data sources available is presented along with examples illustrating cross-facility and internal quality checks required of the data before it should be accepted for validation of LES.

Bridges, James↗