Search NASA⌕ Search

SEARCH · Search NASA

Results for “Metrics and Validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2018

Quantification of Dynamic Model Validation Metrics Using Uncertainty Propagation from Requirements

The Space Launch System, NASA's new large launch vehicle for long range space exploration, is presently in the final design and construction phases, with the first launch scheduled for 2019. A dynamic model of the system has been created and is critical for calculation of interface loads and natural frequencies and mode shapes for guidance, navigation, and control (GNC). Because of the program and schedule constraints, a single modal test of the SLS will be performed while bolted down to the Mobile Launch Pad just before the first launch. A Monte Carlo and optimization scheme will be performed to create thousands of possible models based on given dispersions in model properties and to determine which model best fits the natural frequencies and mode shapes from modal test. However, the question still remains as to whether this model is acceptable for the loads and GNC requirements. An uncertainty propagation and quantification (UP and UQ) technique to develop a quantitative set of validation metrics that is based on the flight requirements has therefore been developed and is discussed in this paper. There has been considerable research on UQ and UP and validation in the literature, but very little on propagating the uncertainties from requirements, so most validation metrics are "rules-of-thumb;" this research seeks to come up with more reason-based metrics. One of the main assumptions used to achieve this task is that the uncertainty in the modeling of the fixed boundary condition is accurate, so therefore that same uncertainty can be used in propagating the fixed-test configuration to the free-free actual configuration. The second main technique applied here is the usage of the limit-state formulation to quantify the final probabilistic parameters and to compare them with the requirements. These techniques are explored with a simple lumped spring-mass system and a simplified SLS model. When completed, it is anticipated that this requirements-based validation metric will provide a quantified confidence and probability of success for the final SLS dynamics model, which will be critical for a successful launch program, and can be applied in the many other industries where an accurate dynamic model is required.

Brown, Andrew M.↗
NASA NTRS2010

ISS Logistics Hardware Disposition and Metrics Validation

I was assigned to the Logistics Division of the International Space Station (ISS)/Spacecraft Processing Directorate. The Division consists of eight NASA engineers and specialists that oversee the logistics portion of the Checkout, Assembly, and Payload Processing Services (CAPPS) contract. Boeing, their sub-contractors and the Boeing Prime contract out of Johnson Space Center, provide the Integrated Logistics Support for the ISS activities at Kennedy Space Center. Essentially they ensure that spares are available to support flight hardware processing and the associated ground support equipment (GSE). Boeing maintains a Depot for electrical, mechanical and structural modifications and/or repair capability as required. My assigned task was to learn project management techniques utilized by NASA and its' contractors to provide an efficient and effective logistics support infrastructure to the ISS program. Within the Space Station Processing Facility (SSPF) I was exposed to Logistics support components, such as, the NASA Spacecraft Services Depot (NSSD) capabilities, Mission Processing tools, techniques and Warehouse support issues, required for integrating Space Station elements at the Kennedy Space Center. I also supported the identification of near-term ISS Hardware and Ground Support Equipment (GSE) candidates for excessing/disposition prior to October 2010; and the validation of several Logistics Metrics used by the contractor to measure logistics support effectiveness.

Rogers, Toneka R.↗
NASA NTRS2022

Combining Machine Learning and Numerical Simulation for High-Resolution PM2.5 Concentration Forecast

Forecasting ambient PM2.5 concentrations with spatiotemporal coverage is key to alerting decision-makers of pollution episodes and preventing detrimental public exposure, especially in regions with limited ground air monitoring stations. The existing methods either rely on chemical transport models (CTMs) to forecast spatial distribution of PM2.5 with nontrivial uncertainty or statistical algorithms to forecast PM2.5 concentration time-series at air monitoring locations without continuous spatial coverage. In this study, we developed a PM2.5 forecast framework by combining the robust Random Forest algorithm with a publicly accessible global CTM forecast product – NASA’s Goddard Earth Observing System “Composition Forecasting” (GEOS-CF), providing spatiotemporally continuous PM2.5 concentration forecasts for the next five days at a 1-km spatial resolution. Our forecast experiment was conducted for a region in Central China including the populous and polluted Fenwei Plain. The forecast for the next two days had overall validation R2 of 0.76 and 0.64, respectively; the R2 was around 0.5 for the following three forecast days. Spatial cross-validation showed similar validation metrics. Our forecast model, with validation normalized mean bias close to zero, substantially reduced the large biases in GEOS-CF. The proposed framework requires minimal computational resources compared to running CTMs at urban scales, enabling near-real-time PM2.5 forecast in resource-restricted environments.

PM2.5↗
NASA NTRS2009

Disruption Tolerant Networking Flight Validation Experiment on NASA's EPOXI Mission

In October and November of 2008, the Jet Propulsion Laboratory installed and tested essential elements of Delay/Disruption Tolerant Networking (DTN) technology on the Deep Impact spacecraft. This experiment, called Deep Impact Network Experiment (DINET), was performed in close cooperation with the EPOXI project which has responsibility for the spacecraft. During DINET some 300 images were transmitted from the JPL nodes to the spacecraft. Then they were automatically forwarded from the spacecraft back to the JPL nodes, exercising DTN's bundle origination, transmission, acquisition, dynamic route computation, congestion control, prioritization, custody transfer, and automatic retransmission procedures, both on the spacecraft and on the ground, over a period of 27 days. All transmitted bundles were successfully received, without corruption. The DINET experiment demonstrated DTN readiness for operational use in space missions. This activity was part of a larger NASA space DTN development program to mature DTN to flight readiness for a wide variety of mission types by the end of 2011. This paper describes the DTN protocols, the flight demo implementation, validation metrics which were created for the experiment, and validation results.

DTN↗
NASA NTRS2020

An Introduction to the NASA GMAO Coupled Atmosphere-Ocean System - GEOS-S2S Version 3

Recently NASA's Global Modeling and Assimilation Office (GMAO) has developed a new Subseasonal to Seasonal Prediction system Version 3 (GEOS-S2S-3). This upgrade replaces the GEOS-S2S-2 which is NASA's current contribution to the North American Multi-Model Experiment seasonal prediction project (Kirtman et al., 2014). The main improvements for our S2S-3 system include 1) a higher resolution MOM5 (Griffies et al., 2005) ocean model (now 0.25o x 0.25o x 50 layers), 2) an improved atmospheric/ocean interface layer (Akella and Suarez, 2018), and 3) assimilation of a long-track satellite salinity into the ocean model (Hackert et al, 2019). Atmospheric forcing is provided by the NASA MERRA-2 reanalysis (Gelaro et al., 2017). Initialization for the ocean relies on the GMAO ocean reanalysis system which assimilates all available in situ temperature and salinity, satellite sea surface salinity, and sea level using the Local Ensemble Transform Kalman Filter (LETKF) implementation of (Penny et al., 2013) on a 5 day assimilation cycle with 20 fixed ensemble members.In this presentation, we will authenticate our new S2S-3 ocean reanalysis using standard GODAE validation metrics. For example, we will compare gridded fields of mean and standard deviation of the ocean reanalysis versus observed fields. We will show correlation/RMS of model versus observations and temperature and salinity mean profiles for the various basins and latitude bands. Basin-scale volume transports, such as the Atlantic Meridional Overturning Circulation and the Indonesian Throughflow will be validated. Equatorial ocean waves will be compared by decomposing sea level into Kelvin and Rossby components. For each of these metrics, we plan to validate the results and then compare our new S2S-3 against the current production version, S2S-2. Finally, we will compare 9-month seasonal forecasts initialized from these two systems for the tropical Pacific NINO3.4 region over the period 1981-present.

Molod, Andrea↗
NASA NTRS2011

Validating LES for Jet Aeroacoustics

Engineers charged with making jet aircraft quieter have long dreamed of being able to see exactly how turbulent eddies produce sound and this dream is now coming true with the advent of large eddy simulation (LES). Two obvious challenges remain: validating the LES codes at the resolution required to see the fluid-acoustic coupling, and the interpretation of the massive datasets that are produced. This paper addresses the former, the use of advanced experimental techniques such as particle image velocimetry (PIV) and Raman and Rayleigh scattering, to validate the computer codes and procedures used to create LES solutions. This paper argues that the issue of accuracy of the experimental measurements be addressed by cross-facility and cross-disciplinary examination of modern datasets along with increased reporting of internal quality checks in PIV analysis. Further, it argues that the appropriate validation metrics for aeroacoustic applications are increasingly complicated statistics that have been shown in aeroacoustic theory to be critical to flow-generated sound, such as two-point space-time velocity correlations. A brief review of data sources available is presented along with examples illustrating cross-facility and internal quality checks required of the data before it should be accepted for validation of LES.

Bridges, James↗
NASA NTRS2023

Atmospheric Correction Inter-comparison eXercise, ACIX-II Land: An Assessment of Amospheric Correction Processors for Landsat 8 and Sentinel-2 Over Land

The correction of the atmospheric effects on optical satellite images is essential for quantitative and multi-temporal remote sensing applications. In order to study the performance of the state-of-the-art methods in an integrated way, a voluntary and open-access benchmark Atmospheric Correction Inter-comparison eXercise (ACIX) was initiated in 2016 in the frame of Committee on Earth Observation Satellites (CEOS) Working Group on Calibration & Validation (WGCV). The first exercise was extended in a second edition wherein twelve atmospheric correction (AC) processors, a substantially larger testing dataset and additional validation metrics were involved. The sites for the inter-comparison analysis were defined by investigating the full catalogue of the Aerosol Robotic Network (AERONET) sites for coincident measurements with satellites' overpass. Although there were more than one hundred sites for Copernicus Sentinel-2 and Landsat 8 acquisitions, the analysis presented in this paper concerns only the common matchups amongst all processors, reducing the number to 79 and 62 sites respectively. Aerosol Optical Depth (AOD) and Water Vapour (WV) retrievals were consequently validated based on the available AERONET observations. The processors mostly succeeded in retrieving AOD for relatively light to medium aerosol loading (AOD < 0.2) with uncertainties <0.08, while the overall uncertainty values were typically 0.23 ± 0.15. Better performances were observed for WV retrievals with >90% of the results falling within the suggested empirical specifications and with the Root Mean Square Error (RMSE) being mostly <0.25 g/cm2. Regarding Surface Reflectance (SR) validation two main approaches were followed. For the first one, a simulated SR reference dataset was computed over all of the test sites by using the 6SV (Second Simulation of the Satellite Signal in the Solar Spectrum vector code) full radiative transfer modelling (RTM) and AERONET measurements for the required aerosol variables and water vapour content. The performance assessment demonstrated that the retrievals were not biased for most of the bands. The uncertainties ranged from approximately 0.003 to 0.01 (excluding B01) for the best performing processors in both sensors' analyses. For the second one, measurements from the radiometric calibration network RadCalNet over La Crau (France) and Gobabeb (Namibia) were involved in the validation. The performance of the processors was in general consistent across all bands for both sensors and with low standard deviations (<0.04) between on-site and estimated surface reflectance. Overall, our study provides a good insight of AC algorithms' performance to developers and users, pointing out similarities and differences for AOD, WV and SR retrievals. Such validation though still lacks of ground-based measurements of known uncertainty to better assess and characterize the uncertainties in SR retrievals.

Atmospheric correction↗
NASA NTRS1987

Methodology requirements for intelligent systems architecture

The methodology required for the development of the 'intelligent system architecture' of distributed computer systems which integrate standard data processing capabilities with symbolic processing to provide powerful and highly autonomous adaptive processing capabilities must encompass three elements: (1) a design knowledge capture system, (2) computer-aided engineering, and (3) verification and validation metrics and tests. Emphasis must be put on the earliest possible definition of system requirements and the realistic definition of allowable system uncertainties. Methodologies must also address human factor issues.

Grant, Terry↗
NASA NTRS2011

Validating LES for Jet Aeroacoustics

Engineers charged with making jet aircraft quieter have long dreamed of being able to see exactly how turbulent eddies produce sound and this dream is now coming true with the advent of large eddy simulation (LES). Two obvious challenges remain: validating the LES codes at the resolution required to see the fluid-acoustic coupling, and the interpretation of the massive datasets that result in having dreams come true. This paper primarily addresses the former, the use of advanced experimental techniques such as particle image velocimetry (PIV) and Raman and Rayleigh scattering, to validate the computer codes and procedures used to create LES solutions. It also addresses the latter problem in discussing what are relevant measures critical for aeroacoustics that should be used in validating LES codes. These new diagnostic techniques deliver measurements and flow statistics of increasing sophistication and capability, but what of their accuracy? And what are the measures to be used in validation? This paper argues that the issue of accuracy be addressed by cross-facility and cross-disciplinary examination of modern datasets along with increased reporting of internal quality checks in PIV analysis. Further, it is argued that the appropriate validation metrics for aeroacoustic applications are increasingly complicated statistics that have been shown in aeroacoustic theory to be critical to flow-generated sound.

Bridges, James↗
NASA NTRS2014

GCR Environmental Models III: GCR Model Validation and Propagated Uncertainties in Effective Dose

This is the last of three papers focused on quantifying the uncertainty associated with galactic cosmic rays (GCR) models used for space radiation shielding applications. In the first paper, it was found that GCR ions with Z>2 and boundary energy below 500 MeV/nucleon induce less than 5% of the total effective dose behind shielding. This is an important finding since GCR model development and validation have been heavily biased toward Advanced Composition Explorer/Cosmic Ray Isotope Spectrometer measurements below 500 MeV/nucleon. Weights were also developed that quantify the relative contribution of defined GCR energy and charge groups to effective dose behind shielding. In the second paper, it was shown that these weights could be used to efficiently propagate GCR model uncertainties into effective dose behind shielding. In this work, uncertainties are quantified for a few commonly used GCR models. A validation metric is developed that accounts for measurements uncertainty, and the metric is coupled to the fast uncertainty propagation method. For this work, the Badhwar-O'Neill (BON) 2010 and 2011 and the Matthia GCR models are compared to an extensive measurement database. It is shown that BON2011 systematically overestimates heavy ion fluxes in the range 0.5-4 GeV/nucleon. The BON2010 and BON2011 also show moderate and large errors in reproducing past solar activity near the 2000 solar maximum and 2010 solar minimum. It is found that all three models induce relative errors in effective dose in the interval [-20%, 20%] at a 68% confidence level. The BON2010 and Matthia models are found to have similar overall uncertainty estimates and are preferred for space radiation shielding applications.

Slaba, Tony C.↗
NASA NTRS2014

Summary and Statistical Analysis of the First AIAA Sonic Boom Prediction Workshop

A summary is provided for the First AIAA Sonic Boom Workshop held 11 January 2014 in conjunction with AIAA SciTech 2014. Near-field pressure signatures extracted from computational fluid dynamics solutions are gathered from nineteen participants representing three countries for the two required cases, an axisymmetric body and simple delta wing body. Structured multiblock, unstructured mixed-element, unstructured tetrahedral, overset, and Cartesian cut-cell methods are used by the participants. Participants provided signatures computed on participant generated and solution adapted grids. Signatures are also provided for a series of uniformly refined workshop provided grids. These submissions are propagated to the ground and loudness measures are computed. This allows the grid convergence of a loudness measure and a validation metric (dfference norm between computed and wind tunnel measured near-field signatures) to be studied for the first time. Statistical analysis is also presented for these measures. An optional configuration includes fuselage, wing, tail, flow-through nacelles, and blade sting. This full configuration exhibits more variation in eleven submissions than the sixty submissions provided for each required case. Recommendations are provided for potential improvements to the analysis methods and a possible subsequent workshop.

Park, Michael A.↗
NASA NTRS2018

Estimation of Lower-Body Kinetics from Loading Profile and Kinematics Alone, Without Measured Ground Reaction Forces

Biomechanical models of human motion can estimate kinetic outcomes, such as joint moments, joint forces and muscle forces. Typically, one performs an inverse dynamics (ID) analysis to compute joint moments from joint angles and measured external forces. Sometimes it is impractical to measure ground reaction forces and moments (GRF&M). We devised an empirical method for performing ID analysis of resistance exercises without measured GRF&M. The method solves the multibody dynamics equations of motion with four key assumptions about the GRF&M that reduce the number of unknowns. The assumptions are 1) negligible ground reaction moments, 2) fixed lateral/medial location of the center of pressure (COP), 3) equal fore/aft location of the COP between the feet, and 4) constant angle of the GRF vector relative to the vertical axis in the frontal plane. We used evaluation trials from a spaceflight countermeasure resistance training device to test this approach. Four participants performed squat and deadlift exercises at various loads. We compared results from traditional ID analysis to results without measured GRF&M using our method. We found that joint moment trajectories in the sagittal plane were qualitatively similar in shape between the two methods, and the amount of root mean squared error (RMSE), measured by difference in joint moment impulse, was typically under 10 percent. Non-sagittal joint moment trajectories, which are much lower in overall magnitude, were not qualitatively similar in shape between the two methods. Non-sagittal moments displayed much higher RMSE, with typical values well over 50 percent. These findings were further supported by validation metrics (Sprague and Geers' P and M metrics, Pearson's r correlation coefficient). Based on these findings, we concluded that useful kinetic results are obtained from ID analysis of squat and deadlift exercises, even when GRF&M are not measured, as long as the outcomes of interest lie in the sagittal plane.

computational modeling↗
NASA NTRS2018

HZETRN Radiation Transport Validation Using Balloon-Based Experimental Data

The deterministic radiation transport code HZETRN (High charge (Z) and Energy TRaNsport) was developed by NASA to study the effects of cosmic radiation on astronauts and instrumentation shielded by various materials. This work presents an analysis of computed differential flux from HZETRN compared with measurement data from three balloon-based experiments over a range of atmospheric depths, particle types, and energies. Model uncertainties were quantified using an interval-based validation metric that takes into account measurement uncertainty both in the flux and the energy at which it was measured. Average uncertainty metrics were computed for the entire dataset as well as subsets of the measurements (by experiment, particle type, energy, etc.) to reveal any specific trends of systematic over- or under-prediction by HZETRN. The distribution of individual model uncertainties was also investigated to study the range and dispersion of errors beyond just single scalar and interval metrics. The differential fluxes from HZETRN were generally well-correlated with balloon-based measurements; the median relative model difference across the entire dataset was determined to be 30%. The distribution of model uncertainties, however, revealed that the range of errors was relatively broad, with approximately 30% of the uncertainties exceeding ± 40%. The distribution also indicated that HZETRN systematically under-predicts the measurement dataset as a whole, with approximately 80% of the relative uncertainties having negative values. Instances of systematic bias for subsets of the data were also observed, including a significant underestimation of alpha particles and protons for energies below 2.5 GeV/u. Muons were found to be systematically over-predicted at atmospheric depths deeper than 50 g/cm(sup 2) but under-predicted for shallower depths. Furthermore, a systematic under-prediction of alpha particles and protons was observed below the geomagnetic cutoff, suggesting that improvements to the light ion production cross sections in HZETRN should be investigated.

James E Warner↗
NASA NTRS2019

Downscaling and Validation of SMAP Radiometer Soil Moisture in CONUS

The SMAP (Soil Moisture Active/Passive) satellite provides global soil moisture (SM) estimates that can be used for scientific research and applications (such as the hydrological cycle, agriculture, ecology, and land atmosphere interactions). Currently, SMAP provides the enhanced radiometer-only SM product (L2SMP) at 9 km grid resolution. However, this spatial resolution is still not enough to satisfy the needs of some studies that require a finer spatial resolution SM product, particularly in agricultural and watershed applications. This study applied a downscaling algorithm to the SMAP 9 km SM product to produce a 1 km resolution over the CONUS (Contiguous United States). The downscaling algorithm is based on the relationship between temperature change and SM modulated by Normalized Difference Vegetation Index (NDVI) of a given time period. This relationship was modeled using variables derived from NLDAS (North America Land Data Assimilation System) and NASA's LTDR (Land Long Term Data Record) between 1981-2018. The algorithm was implemented uses the 1 km MODIS Aqua LST (Land Surface Temperature) product. The downscaled SMAP 1 km SM was validated using in situ SM measurements from the ISMN (International Soil Moisture Network). The validation metrics show an improved overall accuracy of the downscaled SM.

downscaling↗
NASA NTRS2020

Calibration Probe Uncertainty and Validation for the Hypersonic Material Environmental Test System

This paper presents an uncertainty analysis of the stagnation-point calibration probe surface predictions for conditions that span the performance envelope of the Hypersonic Materials Environmental Test System facility located at NASA Langley Research Center. A second-order stochastic expansion was constructed over 47 uncertain parameters to evaluate the sensitivities, identify the most significant uncertain variables, and quantify the uncertainty in the stagnation-point heat flux and pressure predictions of the calibration probe for a low- and high-enthalpy test condition. A sensitivity analysis showed that measurement bias uncertainty is the most significant contributor to the stagnation-point pressure and heat flux variance for the low-enthalpy condition. For the high-enthalpy condition, a paradigm shift in sensitivities revealed the computational fluid dynamics model input uncertainty as the main contributor. A comparison between the prediction and measurement of the stagnation-point conditions under uncertainty showed that there was evidence of statistical disagreement. A validation metric was proposed and applied to the prediction uncertainty to account for the statistical disagreement when compared to the possible stagnation-point heat flux and pressure measurements.

Brune, Andrew J.↗
NASA NTRSDate not supplied

Development and Initial Validation of HFBP-EM Short-Form Surveys

The Human Factors and Behavioral Performance Exploration Measures (HFBP-EM) suite includes self-report surveys designed to assess Behavioral Medicine (BMed) and Team risks for future exploration class missions. Survey data was collected during the Human Exploration Research Analog (HERA) Campaign 4 (C4), HERA Campaign 5 (C5), and during SIRIUS19 missions in the Russian Ground Based Experiment Complex, NEK. Crew time in spaceflight analogs and in spaceflight is typically limited. Extensive self-report surveys can be a significant burden to participants and are not operationally acceptable for spaceflight. The purpose of this project was to comprehensively assess the reliability and validity of the standard-length HFBP-EM surveys, identify acceptable short forms to administer during HERA C6, examine initial validation evidence for the proposed short form measures, and identify key measurement gaps in HFBP-EM coverage. In this presentation, we will report reliability and validity metrics of standard form data, describe the proposed new short forms, and present initial psychometric analyses supporting the use of these short forms.

S T Bell↗
NASA NTRS1983

Metric analysis and data validation across FORTRAN projects

The desire to predict the effort in developing or explaining the quality of software has led to the proposal of several metrics. As a step toward validating these metrics, the Software Engineering Laboratory (SEL) has analyzed the software science metrics, cyclomatic complexity, and various standard program measures for their relation to effort (including design through acceptance testing), development errors (both discrete and weighted according to the amount of time to locate and fix), and one another. The data investigated are collected from a project FORTRAN environment and examined across several projects at once, within individual projects and by reporting accuracy checks demonstrating the need to validate a database. When the data comes from individual programmers or certain validated projects, the metrics' correlations with actual effort seem to be strongest. For modules developed entirely by individual programmers, the validity ratios induce a statistically significant ordering of several of the metrics' correlations. When comparing the strongest correlations, neither software science's E metric cyclomatic complexity not source lines of code appears to relate convincingly better with effort than the others.

Basili, Victor R.↗