Search NASA⌕ Search

SEARCH · Search NASA

Results for “statistical metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

Kepler Planet Detection Metrics: Statistical Bootstrap Test

This document describes the data produced by the Statistical Bootstrap Test over the final three Threshold Crossing Event (TCE) deliveries to NExScI: SOC 9.1 (Q1Q16)1 (Tenenbaum et al. 2014), SOC 9.2 (Q1Q17) aka DR242 (Seader et al. 2015), and SOC 9.3 (Q1Q17) aka DR253 (Twicken et al. 2016). The last few years have seen significant improvements in the SOC science data processing pipeline, leading to higher quality light curves and more sensitive transit searches. The statistical bootstrap analysis results presented here and the numerical results archived at NASAs Exoplanet Science Institute (NExScI) bear witness to these software improvements. This document attempts to introduce and describe the main features and differences between these three data sets as a consequence of the software changes.

Bootstrap↗
NASA NTRS2005

INDUCTIVE SYSTEM HEALTH MONITORING WITH STATISTICAL METRICS

Model-based reasoning is a powerful method for performing system monitoring and diagnosis. Building models for model-based reasoning is often a difficult and time consuming process. The Inductive Monitoring System (IMS) software was developed to provide a technique to automatically produce health monitoring knowledge bases for systems that are either difficult to model (simulate) with a computer or which require computer models that are too complex to use for real time monitoring. IMS processes nominal data sets collected either directly from the system or from simulations to build a knowledge base that can be used to detect anomalous behavior in the system. Machine learning and data mining techniques are used to characterize typical system behavior by extracting general classes of nominal data from archived data sets. In particular, a clustering algorithm forms groups of nominal values for sets of related parameters. This establishes constraints on those parameter values that should hold during nominal operation. During monitoring, IMS provides a statistically weighted measure of the deviation of current system behavior from the established normal baseline. If the deviation increases beyond the expected level, an anomaly is suspected, prompting further investigation by an operator or automated system. IMS has shown potential to be an effective, low cost technique to produce system monitoring capability for a variety of applications. We describe the training and system health monitoring techniques of IMS. We also present the application of IMS to a data set from the Space Shuttle Columbia STS-107 flight. IMS was able to detect an anomaly in the launch telemetry shortly after a foam impact damaged Columbia's thermal protection system.

Iverson, David L.↗
NASA NTRS2018

Overview of Metrics Pertinent to Human Response to UAM Noise

This presentation reviews metrics pertinent to noise from eVTOL aircraft, including Noise dose metrics; DNL, CTL; Statistical metrics; Spectrum-based metrics; metrics based on event frequency; and metrics based on partial loudness and detection.

aircraft noise disturbance↗
NASA NTRS2022

Evaluation of remote sensing-based evapotranspiration products at low-latitude eddy covariance sites

Remote sensing-based evapotranspiration (ET) products have been evaluated primarily using data from northern middle latitudes; therefore, little is known about their performance at low latitudes. To address this bias, an evaluation dataset was compiled using eddy covariance data from 40 sites between latitudes 30° S and 30° N. The flux data were obtained from the emerging network in Mexico (MexFlux) and from openly available databases of FLUXNET, AsiaFlux, and OzFlux. This unique reference dataset was then used to evaluate remote sensing-based ET products in environments that have been underrepresented in earlier studies. The evaluated products were: MODIS ET (MOD16, both the discontinued collection 5 (C5) and the latest collection (C6)), Global Land Evaporation Amsterdam Model (GLEAM) ET, and Atmosphere-Land Exchange Inverse (ALEXI) ET. Products were compared with unadjusted fluxes (ETorig) and with fluxes corrected for the lack of energy balance closure (ETebc). Three common statistical metrics were used: coefficient of determination (R2), root mean square error (RMSE), and percent bias (PBIAS). The effect of a vegetation mismatch between pixel and site on product evaluation results was investigated by examining the relationship between the statistical metrics and product-specific vegetation match indexes. Evaluation results of this study and those published in the literature were used to examine the performance of the products across latitudes. Differences between the MOD16 collection 5 and 6 datasets were generally smaller than differences with the other products. Performance and ranking of the evaluated products depended on whether ETorig or ETebc was used. When using ETorig, GLEAM generally had the highest R2, smallest PBIAS, and best RMSE values across the studied land cover types and climate zones. Neither MOD16 nor ALEXI performed consistently better than the other. When using ETebc, none of the products stood out in terms of both low bias and strong correlations. The use of ETebc instead of ETorig affected the biases more than the correlations. The product evaluation results showed no significant relationship with the degree of match between the vegetation at the pixel and site scale. The latitudinal comparison showed tendencies of lower R2 (all products) but better PBIAS and normalized RMSE values (MOD16 and GLEAM) for forests at low latitudes than for forests at northern middle latitudes. For non-forest vegetation, the products showed no clear latitudinal differences in performance.

Diego Salazar-Martínez↗
NASA NTRS2016

Performance Metrics, Error Modeling, and Uncertainty Quantification

A common set of statistical metrics has been used to summarize the performance of models or measurements-­ the most widely used ones being bias, mean square error, and linear correlation coefficient. They assume linear, additive, Gaussian errors, and they are interdependent, incomplete, and incapable of directly quantifying un­certainty. The authors demonstrate that these metrics can be directly derived from the parameters of the simple linear error model. Since a correct error model captures the full error information, it is argued that the specification of a parametric error model should be an alternative to the metrics-based approach. The error-modeling meth­odology is applicable to both linear and nonlinear errors, while the metrics are only meaningful for linear errors. In addition, the error model expresses the error structure more naturally, and directly quantifies uncertainty. This argument is further explained by highlighting the intrinsic connections between the performance metrics, the error model, and the joint distribution between the data and the reference.

Quantification↗
NASA NTRS2018

Performance Metrics for the Assessment of Satellite Data Products: An Ocean Color Case Study

Performance assessment of ocean color satellite data has generally relied on statistical metrics chosen for their common usage and the rationale for selecting certain metrics is infrequently explained. Commonly reported statistics based on mean squared errors, such as the coefficient of determination (r2), root mean square error, and regression slopes, are most appropriate for Gaussian distributions without outliers and, therefore, are often not ideal for ocean color algorithm performance assessment, which is often limited by sample availability. In contrast, metrics based on simple deviations, such as bias and mean absolute error, as well as pair-wise comparisons, often provide more robust and straightforward quantities for evaluating ocean color algorithms with non-Gaussian distributions and outliers. This study uses a SeaWiFS chlorophyll-a validation data set to demonstrate a framework for satellite data product assessment and recommends a multimetric and user-dependent approach that can be applied within science, modeling, and resource management communities.

remote sensing↗
NASA NTRS2018

A Blueprint for Demonstrating Quantum Supremacy with Superconducting Qubits

Long coherence times and high fidelity control recently achieved in scalable superconducting circuits paved the way for the growing number of experimental studies of many-qubit quantum coherent phenomena in these devices. Albeit full implementation of quantum error correction and fault tolerant quantum computation remains a challenge the near term pre-error correction devices could allow new fundamental experiments despite inevitable accumulation of errors. One such open question foundational for quantum computing is achieving the so called quantum supremacy, an experimental demonstration of a computational task that takes polynomial time on the quantum computer whereas the best classical algorithm would require exponential time and/or resources. It is possible to formulate such a task for a quantum computer consisting of less than a 100 qubits. The computational task we consider is to provide approximate samples from a non-trivial quantum distribution. This is a generalization for the case of superconducting circuits of ideas behind boson sampling protocol for quantum optics introduced by Arkhipov and Aaronson. In this presentation we discuss a proof-of-principle demonstration of such a sampling task on a 9-qubit chain of superconducting gmon qubits developed by Google. We discuss theoretical analysis of the driven evolution of the device resulting in output approximating samples from a uniform distribution in the Hilbert space, a quantum chaotic state. We analyze quantum chaotic characteristics of the output of the circuit and the time required to generate a sufficiently complex quantum distribution. We demonstrate that the classical simulation of the sampling output requires exponential resources by connecting the task of calculating the output amplitudes to the sign problem of the Quantum Monte Carlo method. We also discuss the detailed theoretical modeling required to achieve high fidelity control and calibration of the multi-qubit unitary evolution in the device. We use a novel cross-entropy statistical metric as a figure of merit to verify the output and calibrate the device controls. Finally, we demonstrate the statistics of the wave function amplitudes generated on the 9-gmon chain and verify the quantum chaotic nature of the generated quantum distribution. This verifies the implementation of the quantum supremacy protocol.

Kechedzhi, Kostyantyn↗
NASA NTRS2006

Use of High-resolution WRF Simulations to Forecast Lightning Threat

Recent observational studies have confirmed the existence of a robust statistical relationship between lightning flash rates and the amount of large precipitating ice hydrometeors in storms. This relationship is exploited, in conjunction with the capabilities of recent forecast models such as WRF, to forecast the threat of lightning from convective storms using the output fields from the model forecasts. The simulated vertical flux of graupel at -15C is used in this study as a proxy for charge separation processes and their associated lightning risk. Six-h simulations are conducted for a number of case studies for which three-dimensional lightning validation data from the North Alabama Lightning Mapping Array are available. Experiments indicate that initialization of the WRF model on a 2 km grid using Eta boundary conditions, Doppler radar radial velocity and reflectivity fields, and METAR and ACARS data yield the most realistic simulations. An array of subjective and objective statistical metrics are employed to document the utility of the WRF forecasts. The simulation results are also compared to other more traditional means of forecasting convective storms, such as those based on inspection of the convective available potential energy field.

McCaul, William E.↗
NASA NTRS2007

A Comparison of Two Balance Calibration Model Building Methods

Simulated strain-gage balance calibration data is used to compare the accuracy of two balance calibration model building methods for different noise environments and calibration experiment designs. The first building method obtains a math model for the analysis of balance calibration data after applying a candidate math model search algorithm to the calibration data set. The second building method uses stepwise regression analysis in order to construct a model for the analysis. Four balance calibration data sets were simulated in order to compare the accuracy of the two math model building methods. The simulated data sets were prepared using the traditional One Factor At a Time (OFAT) technique and the Modern Design of Experiments (MDOE) approach. Random and systematic errors were introduced in the simulated calibration data sets in order to study their influence on the math model building methods. Residuals of the fitted calibration responses and other statistical metrics were compared in order to evaluate the calibration models developed with different combinations of noise environment, experiment design, and model building method. Overall, predicted math models and residuals of both math model building methods show very good agreement. Significant differences in model quality were attributable to noise environment, experiment design, and their interaction. Generally, the addition of systematic error significantly degraded the quality of calibration models developed from OFAT data by either method, but MDOE experiment designs were more robust with respect to the introduction of a systematic component of the unexplained variance.

DeLoach, Richard↗
NASA NTRS2007

High-Resolution WRF Forecasts of Lightning Threat

Tropical Rainfall Measuring Mission (TRMM)lightning and precipitation observations have confirmed the existence of a robust relationship between lightning flash rates and the amount of large precipitating ice hydrometeors in storms. This relationship is exploited, in conjunction with the capabilities of the Weather Research and Forecast (WRF) model, to forecast the threat of lightning from convective storms using the output fields from the model forecasts. The simulated vertical flux of graupel at -15C is used in this study as a proxy for charge separation processes and their associated lightning risk. Initial experiments using 6-h simulations are conducted for a number of case studies for which three-dimensional lightning validation data from the North Alabama Lightning Mapping Array are available. The WRF has been initialized on a 2 km grid using Eta boundary conditions, Doppler radar radial velocity and reflectivity fields, and METAR and ACARS data. An array of subjective and objective statistical metrics is employed to document the utility of the WRF forecasts. The simulation results are also compared to other more traditional means of forecasting convective storms, such as those based on inspection of the convective available potential energy field.

Goodman, S. J.↗
NASA NTRS2010

Advanced Response Surface Modeling of Ares I Roll Control Jet Aerodynamic Interactions

The Ares I rocket uses roll control jets. These jets have aerodynamic implications as they impinge on the surface and protuberances of the vehicle. The jet interaction on the body can cause an amplification or a reduction of the rolling moment produced by the jet itself, either increasing the jet effectiveness or creating an adverse effect. A design of experiments test was planned and carried out using computation fluid dynamics, and a subsequent response surface analysis ensued on the available data to characterize the jet interaction across the ascent portion of the Ares I flight envelope. Four response surface schemes were compared including a single response surface covering the entire design space, separate sector responses that did not overlap, continuously overlapping surfaces, and recursive weighted response surfaces. These surfaces were evaluated on traditional statistical metrics as well as visual inspection. Validation of the recursive weighted response surface was performed using additionally available data at off-design point locations.

Favaregh, Noah M.↗
NASA NTRS2016

Effect of Stagger on the Vibroacoustic Loads from Clustered Rockets

The effect of stagger startup on the vibro-acoustic loads that form during the end- effects-regime of clustered rockets is studied using both full-scale (hot-gas) and laboratory scale (cold gas) data. Both configurations comprise three nozzles with thrust optimized parabolic contours that undergo free shock separated flow and restricted shock separated flow as well as an end-effects regime prior to flowing full. Acoustic pressure waveforms recorded at the base of the nozzle clusters are analyzed using various statistical metrics as well as time-frequency analysis. The findings reveal a significant reduction in end- effects-regime loads when engine ignition is staggered. However, regardless of stagger, both the skewness and kurtosis of the acoustic pressure time derivative elevate to the same levels during the end-effects-regime event thereby demonstrating the intermittence and impulsiveness of the acoustic waveforms that form during engine startup.

Rojo, Raymundo↗
NASA NTRS2019

Evaluation of the Impacts of Biomass Burning Aerosols on the NASA GEOS Sub-Seasonal Climate Forecasts

Emission sources of trace gases and aerosol particles in the South American (SA)and African (Af) continents have a strong seasonal and space variability associated with the extensive vegetation fires activities. In both continents, during the austral winter, the fires affect mainly tropical forest and savannah-type biomes and are mostly associated with deforestation and agricultural/pasture land management. Smoke aerosol particles, on average, contribute to at least 90% of the total aerosol optical depth (AOD) in the visible spectrum in the case of the South America regional smoke. Smoke aerosols also act as cloud condensation nuclei affecting cloud microphysics properties and therefore, changing the radiation budget, hydrological cycle and global circulation patterns over disturbed areas (Kaufman, 1995; Rosenfeld, 1999; Andreae,et al., 2004; Koren et al., 2004, Zhang, 2008; Ott et al., 2010; Randles et al., 2013). This study aims to evaluate and quantify the impact of including a comprehensive emission field of biomass burning aerosol on the performance of a seasonal climate forecast system, not only regarding the AOD itself but mainly on the meteorological state variable (e.g., precipitation and temperature). To address the questions put above, we designed two numerical experiments: 1- named"AERO_CTL" which applies the Quick Fire Emissions Dataset (QFED) emissions estimated with intra-diurnal variation (hereafter, BBE), and 2- named "AERO_CLM" where the sourcee mission is based on a climatology of the QFED emissions, with only monthly variation(hereafter, BBCLIM). Hindcast simulations were produced using the Goddard Earth ObservingSystem global circulation model, version 5, sub-seasonal to seasonal (GEOS5-S2S) system with a nominal spatial resolution of 56km (Rienecker et al., 2008). In both experiments, the aerosol feedbacks from cloud developments and radiation interactions were accounted. The two experiments consisted of 4 members each and ran from June to November spanning over the years 2000 to 2015. Model performance was evaluated by calculating statistical metrics on the mean area of SA and Af. Our results demonstrated that the skill model in predicting AOD is significantly improve when BBE source emission is applied over SA, but not over the Afcontinent. Over SA, the correlation between the AERO_CTL model configuration and MERRA-2 is 0.93 (R2= 0.86, RMS=0.02, BIAS=0.01), while the AERO_CLM model presents a value of0.81 (R2= 0.65, RMS=0.04, BIAS=0.06). However, the AERO_CTL experiment better represents the inter-annual variability of the AOS in both regions. The gain of the skill in predicting the AOD by the AERO_CTL experiment is also seen in some meteorological variables. We observed an increase in the model skill in predicting the 2-meter temperature and precipitation of up to 0.3 for the AERO_CTL experiment in comparison to the AERO_CLM. AERO_CLM. According to the analyzed hindcast, we inferred that representing the BBE more realistically implies in a significant gain of skills in the seasonal climate forecasting over SA and Af continents.

Aerosol↗
NASA NTRS2019

Sea Surface Salinity Retrievals from Aquarius Using Neural Networks

Even though the Sea Surface Salinity (SSS) retrieved from Aquarius are generally very close to in-situ measurements, the level of similarity varies with the region and with the circumstances of the observations (wind speed, sea surface temperature, etc.). SSS is currently retrieved from the brightness temperatures measured by Aquarius and applying the current theoretical model for the propagation and emission of the natural thermal radiation. In this contribution we consider an alternative retrieval approach based on a Neural Network (NN) with the goal of improving the subsets of Aquarius SSS data that are in poorer agreement within-situ measurements. The subset considered here are the SSS retrieved at latitudes higher than 30 ̊. The output of the NN approach are compared against in-situ measurements using four statistical metrics (correlation coefficient, bias, RMSD and 5% trimmed range). The output of the NN and the nominal Aquarius SSS are compared against SSS values from in-situ measurements and from ocean models. From these comparisons it appears that the output of the NN matches the in-situ measurements better than the nominal Aquarius SSS.

Soldo, Yan↗
NASA NTRS2020

Ensemble PM2.5 Forecasting during the 2018 Camp Fire Event Using the HYSPLIT Transport and Dispersion Model

Biomass burning releases a vast amount of aerosols into the atmosphere, often leading to severe air quality and health problems. Prediction of the air quality effects from biomass burning emissions is challenging due to uncertainties in fire emission, plume rise calculation, and other model inputs/processes. Ensemble forecasting is increasingly used to represent model uncertainties. In this paper, an ensemble forecast was conducted to predict surface PM2.5 during the 2018 California Camp Fire event using the NOAA HYSPLIT dispersion model at 0.1- degree horizontal resolution. Different combinations of four satellite-based fire emission datasets (FEER, FLAMBE, GBBEPx and GFAS), two plume rise schemes (Briggs and Sofiev), various meteorology inputs and model setup options were used to create the forecast ensemble, for a total of 112 experiments. The performance of each ensemble member and the ensemble mean were evaluated using ground-based observations, with four statistical metrics and an overall rank. The ensemble spread of the 112 members reached 1000 μg/m3, highlighting the large uncertainty in wildfire forecast. The ensemble mean displayed the best performance. Each fire emission product contributed to one or more members among the top ten performers, revealing the forecasting dependence on both the quality of fire emissions data and model representation of emission, transport and removal processes. In addition, an ensemble size reduction technique was introduced. With the help of this technique, the ensemble size was reduced from 112 to 28 members and still produced an ensemble mean that yielded comparable or even better performance to that of the full ensemble.

Biomass Burning↗
NASA NTRS2021

Validation of Satellite-Based Precipitation Products from TRMM to GPM

The global precipitation measurement mission (GPM) has been in operation for seven years and continues to provide a vast quantity of global precipitation data at finer temporospatial resolutions with improved accuracy and coverage. GPM’s signature algorithm, the integrated multisatellite retrievals for GPM (IMERG) is a next-generation of precipitation product expected for wide variety of research and operational applications. This study evaluates the latest version (V06B) of IMERG and its predecessor, the tropical rainfall measuring mission (TRMM) multisatellite precipitation (TMPA) 3B42 (V7) using ground-based and gauge-corrected multiradar multisensor system (MRMS) precipitation products over the conterminous United States (CONUS). The spatial distributions of all products are analyzed. The error characteristics are further examined for 3B42 and IMERG in winter and summer by an error decomposition approach, which partitions total bias into hit bias, biases due to missed precipitation and false precipitation. The volumetric and categorical statistical metrics are used to quantitatively evaluate the performance of the two satellite-based products. All products show a similar precipitation climatology with some regional differences. The two satellite-based products perform better in the eastern CONUS than in the mountainous Western CONUS. The evaluation demonstrates the clear improvement in IMERG precipitation product in comparison with its predecessor 3B42, especially in reducing missed precipitation in winter and summer, and hit bias in winter, resulting in better performance in capturing lighter and heavier precipitation.

MRMS↗
NASA NTRSDate not supplied

Influence of Thermophysical Property Variability on Thermal Predictions of Ti-6Al-4V

The additive manufacturing (AM) industry has experienced rapid growth in recent decades as industrial interest in the process has grown. Because of this, there are many groups interested in simulations of the AM process. However, the influence of thermophysical property variability on the melt pool geometry during processing of various AM alloys is unclear. The goal of my work at NASA Langley Research Center (LaRC) was to characterize this influence on Ti-6Al-4V during AM processing alongside developing a tool to enable equivalent studies on other AM materials. To facilitate this process, a database tool was developed to contain and organize thermophysical data previously reported by primary sources. The specific thermophysical properties of interest for this study were density, specific heat capacity, and conductivity. Thermal diffusivity was calculated using the other three properties. This data was then fit with a Gaussian distribution and then randomly sampled from using Monte Carlo random value sampling. Using the Rosenthal equation, this sample data was used to simulate the temperature field during additive manufacturing and extract the melt pool geometry. Specifically, the melt pool’s length, width, and depth. This process was repeated an arbitrary number of times, with the default being 1000. Once this information was obtained, histograms were made showing the distributions of the sizes of the melt pool’s length, width, and depth. Representative statistical metrics of the distributions were calculated (mean, standard deviation, and coefficient of variance). This work found that at the simulated processing values, the melt pool’s width and depth had a 5.7% variation while the length had a 3.2% variation. Additionally, a tool was constructed to graph a thermal color map representation of the melt pool for specific, arbitrary values of density, specific heat, and conductivity. This tool allowed for more efficient plotting of individual queries of the thermophysical properties. The results that were observed in the research were that values reported in literature vary and this variance can have a significant impact on the simulation. Predictions of the melt pool’s dimensions show that length has a standard deviation of approximately 6 μm, width has a standard deviation of approximately 5 μm, and depth can vary by approximately 3 μm. Considering the mean sizes of length, width, and depth are 185 μm, 88 μm, and 44 μm respectively, such a deviation is significant. This shows that depth, for example, could be more than 10% larger or smaller than expected. This demonstrates that variability in the reported thermophysical properties are not negligible and should be expected to have an influence on the results of the laser powder bed fusion additive manufacturing process. When simulating this process in the future, measures should be taken to account for this discrepancy and the uncertainty involved

Justin Martin↗
NASA NTRS2021

Evaluation, Analysis, and Application of Internal Strain-Gage Balance Data

Experimental processes, analytical methods, and numerical algorithms are described that may be used to predict the forces and moments of an internal strain-gage balance during a wind tunnel test. First, the control volume model of a strain-gage balance and the concepts of load state, load space, and output space are introduced. These important abstractions provide a better understanding of fundamental characteristics of different balance load prediction approaches. Then, the description of strain-gage balance data and the definition of the primary gage sensitivity is discussed. Afterwards, basic elements of the calibration of a typical six-component balance are reviewed. Two fundamentally different balance load prediction methods, the processing of check loads, and related topics are also discussed. Three real-world balance data examples are reviewed in great detail to illustrate typical analysis results for a variety of strain-gage balance designs. Finally, important observations are summarized and recommendations are provided. Additional information and detailed mathematical derivations can be found in the appendices of the document. They include the following topics: balance terminology, definitions of important statistical metrics, balance axis system conventions, balance load transformations, the combined load diagram, electrical output format options, bi-directional gage output characteristics, determination of the natural zeros, derivation of two balance load prediction methods, description of two tare load iteration algorithms, modeling of balance temperature effects, basics of three-component moment balances, definition of the percent contribution, detection of linear and near-linear dependencies in balance calibration data, a regression model term selection algorithm, and other related topics.

wind tunnel test↗