SEARCH · Search NASA

Results for “statistical metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Extreme metrics from large ensembles: investigating the effects of ensemble size on their estimates

Abstract. We consider the problem of estimating the ensemble sizes required to characterize the forced component and the internal variability of a number of extreme metrics. While we exploit existing large ensembles, our perspective is that of a modeling center wanting to estimate a priori such sizes on the basis of an existing small ensemble (we assume the availability of only five members here). We therefore ask if such a small-size ensemble is sufficient to estimate accurately the population variance (i.e., the ensemble internal variability) and then apply a well-established formula that quantifies the expected error in the estimation of the population mean (i.e., the forced component) as a function of the sample size n, here taken to mean the ensemble size. We find that indeed we can anticipate errors in the estimation of the forced component for temperature and precipitation extremes as a function of n by plugging into the formula an estimate of the population variance derived on the basis of five members. For a range of spatial and temporal scales, forcing levels (we use simulations under Representative Concentration Pathway 8.5) and two models considered here as our proof of concept, it appears that an ensemble size of 20 or 25 members can provide estimates of the forced component for the extreme metrics considered that remain within small absolute and percentage errors. Additional members beyond 20 or 25 add only marginal precision to the estimate, and this remains true when statistical inference through extreme value analysis is used. We then ask about the ensemble size required to estimate the ensemble variance (a measure of internal variability) along the length of the simulation and – importantly – about the ensemble size required to detect significant changes in such variance along the simulation with increased external forcings. Using the F test, we find that estimates on the basis of only 5 or 10 ensemble members accurately represent the full ensemble variance even when the analysis is conducted at the grid-point scale. The detection of changes in the variance when comparing different times along the simulation, especially for the precipitation-based metrics, requires larger sizes but not larger than 15 or 20 members. While we recognize that there will always exist applications and metric definitions requiring larger statistical power and therefore ensemble sizes, our results suggest that for a wide range of analysis targets and scales an effective estimate of both forced component and internal variability can be achieved with sizes below 30 members. This invites consideration of the possibility of exploring additional sources of uncertainty, such as physics parameter settings, when designing ensemble simulations.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

The Power of Many: An Ensemble Approach to Spectral Similarity

Quantifying the similarity between two mass spectra─a known reference mass spectrum and an unidentified sample mass spectrum─is at the heart of compound identification workflows in gas chromatography–mass spectrometry (GC-MS). The reference spectrum most like the sample is assigned as its identification (provided some quantitative similarity threshold is met, e.g., 80%) and thus accurately measuring similarity is essential. Significant research has gone toward developing metrics for this purpose, each of which has attempted to improve upon existing methods by incorporating GC-MS-specific information (e.g., peak ratios or retention times) or adopting various statistical and algorithmic frameworks. While this active development has led to a plethora of similarity metrics with demonstrated value across different contexts, the unfortunate consequence has been confusion surrounding which metric should be used as a global standard. No such metric is currently accepted as the standard method because different metrics have demonstrated optimal performance in different contexts. In this work, we propose an ensemble approach to spectral similarity scoring that combines the collective information from across existing similarity metrics to form an improved, globally representative similarity metric as a step toward establishing a global standard method. In conclusion, the resulting ensemble metrics are evaluated on over 88,000 spectra of varying complexity and demonstrate improved abilities to accurately rank the correct reference spectrum as the top-matching candidate for a sample relative to the rankings generated by individual similarity scores.

Carbohydrates↗

DOE OSTI2020

Impacts of Stochastic Mixing in Idealized Convection-Permitting Simulations of Squall Lines

This study investigates impacts of altering subgrid-scale mixing in “convection-permitting” kilometer-scale horizontal-grid-spacing (Δ h ) simulations by applying either constant or stochastic multiplicative factors to the horizontal mixing coefficients within the Weather Research and Forecasting Model. In quasi-idealized 1-km Δ h simulations of two observationally based squall-line cases, constant enhanced mixing produces larger updraft cores that are more dilute at upper levels, weakens the cold pool, rear-inflow jet, and front-to-rear flow of the squall line, and degrades the model’s effective resolution. Reducing mixing by a constant multiplicative factor has the opposite effect on all metrics. Completely turning off parameterized horizontal mixing produces bulk updraft statistics and squall-line mesoscale structure closest to an LES “benchmark” among all 1-km simulations, although the updraft cores are too undilute. The stochastic mixing scheme, which applies a multiplicative factor to the mixing coefficients that varies stochastically in time and space, is employed at 0.5-, 1-, and 2-km Δ h . It generally reduces midlevel vertical velocities and enhances upper-level vertical velocities compared to simulations using the standard mixing scheme, with more substantial impacts at 1- and 2-km Δ h compared to 0.5-km Δ h . Further, the stochastic scheme also increases updraft dilution to better agree with the LES for one case, but has less impact on the other case. Stochastic mixing acts to weaken the cold pool but without a significant impact on squall-line propagation. It also does not affect the model’s overall effective resolution unlike applying constant multiplicative factors to the mixing coefficients.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2023

Validation of LOCA2 and STAR-ESDM Statistically Downscaled Products

The National Climate Assessment (NCA) is the preeminent national report examining current and future risks posed by climate change. Countless agencies, policymakers, stakeholders and other end-users rely upon guidance from the NCA to plan for an uncertain future. These groups all depend on modern curated data, provided alongside the NCA, to quantify the impact of climate change on metrics of relevance for their decision processes. In its fifth iteration (NCA5), two statistically downscaled ensemble products, each providing data at grid spacing of approximately 5km over the contiguous United States, were selected to accompany the report. These include LOCalized Analogs version 2 (LOCA2) and Seasonal Trends and Analysis of Residuals Empirical-Statistical Downscaling Model (STAR-ESDM). Both data products are produced through a process known as statistical downscaling, where relatively coarse Global Climate Model (GCM) data is refined to locally relevant scales through the application of scientifically-supported empirical and algorithmic relationships. In support of the NCA effort, this report provides an independent validation of these two products against historical observations, with a focus on precipitation and near-surface temperature variables. Based on the results of this validation, several recommendations are provided related to the use of these data products. The structure of this report is as follows: In section 2, we review three gridded observational products that are used as part of our intercomparison. In section 3, we describe the two statistical downscaling techniques and their corresponding datasets that are the focus of this study. In section 4, the methodology we employ for validation is described. Section 5 provides results of the validation, which in turn motivate our recommendations on the use of these data products. A brief summary is provided in section 6.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Clutter Distributions for Tomographic Image Standardization in Ground-Penetrating Radar

Multistatic ground-penetrating radar (GPR) signals can be imaged tomographically to produce 3-D distributions of image intensities. In the absence of objects of interest, these intensities can be considered to be estimates of clutter. These clutter intensities spatially vary over several orders of magnitude and vary across different arrays, which makes a direct comparison of these raw intensities difficult. However, by gathering statistics on these intensities and their spatial variation, a variety of metrics can be determined. In this study, the clutter distribution is found to fit better to a two-parameter Weibull distribution than Gaussian or log-normal distributions. Based on the spatial variation of the two Weibull parameters, scale and shape, more information may be gleaned from these data. How well the GPR array is illuminating various parts of the ground, in depth and cross track, may be determined from the spatial variation of the Weibull scale parameter, which may in turn be used to estimate an effective attenuation coefficient in the soil. The transition in depth from clutter- to noise-limited conditions (which is one possible definition of GPR penetration depth) can be estimated from the spatial variation of the Weibull shape parameter. Lastly, the underlying clutter distributions also provide an opportunity to standardize image intensities to determine when a statistically significant deviation from background (clutter) has occurred, which is convenient for buried threat detection algorithm development that needs to be robust across multiple different arrays.

42 ENGINEERING↗

DOE OSTI2024

FedFleet 2024: Fleet Data Quality - Understanding and Improving Your Agency's FAST Fleet Data Quality Metric? [Slides]

This presentation presents an overview of how the team supporting the Federal Automotive Statistical Tool (FAST) project for DOE and GSA view and assess the quality of fleet data submissions from federal agencies. It also discusses ways in which agencies can improve the quality of their fleet data and of their fleet data submissions. FAST is a web-based information system sponsored by GSA's Office of Government-wide Policy and DOE's Federal Energy Management Program to collect information about the US federal government's fleet of motor vehicles; FAST is developed, maintained, and supported by DOE's Idaho National Laboratory (INL).

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

Validation Metrics for Fixed Effects and Mixed-Effects Calibration

The modern scientific process often involves the development of a predictive computational model. To improve its accuracy, a computational model can be calibrated to a set of experimental data. A variety of validation metrics can be used to quantify this process. Some of these metrics have direct physical interpretations and a history of use, while others, especially those for probabilistic data, are more difficult to interpret. In this work, a variety of validation metrics are used to quantify the accuracy of different calibration methods. Frequentist and Bayesian perspectives are used with both fixed effects and mixed-effects statistical models. Through a quantitative comparison of the resulting distributions, the most accurate calibration method can be selected. Two examples are included which compare the results of various validation metrics for different calibration methods. It is quantitatively shown that, in the presence of significant laboratory biases, a fixed effects calibration is significantly less accurate than a mixed-effects calibration. This is because the mixed-effects statistical model better characterizes the underlying parameter distributions than the fixed effects model. The results suggest that validation metrics can be used to select the most accurate calibration model for a particular empirical model with corresponding experimental data.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

The first ensemble of kilometer-scale simulations of a hydrological year over the third pole

An accurate understanding of the current and future water cycle over the Third Pole is of great societal importance, given the role this region plays as a water tower for densely populated areas downstream. An emerging and promising approach for skillful climate assessments over regions of complex terrain is kilometer-scale climate modeling. As a foundational step towards such simulations over the Third Pole, we present a multi-model and multi-physics ensemble of kilometer-scale regional simulations for the hydrological year of October 2019 to September 2020. The ensemble consists of 13 simulations performed by an international consortium of 10 research groups, configured with a horizontal grid spacing ranging from 2.2 to 4 km covering all of the Third Pole region. These simulations are driven by ERA5 and are part of a Coordinated Regional Climate Downscaling EXperiment Flagship Pilot Study on Convection-Permitting Third Pole. The simulations are compared against available gridded and in-situ observations and remote-sensing data, to assess the performance and spread of the model ensemble compared to the driving reanalysis during the cold and warm seasons. Although ensemble evaluation is hindered by large differences between the gridded precipitation datasets used as a reference over this region, we show that the ensemble improves on many warm-season precipitation metrics compared with ERA5, including most wet-day and hour statistics, and also adds value in the representation of wet spells in both seasons. As such, the ensemble will provide an invaluable resource for future improvements in the process understanding of the hydroclimate of this remote but important region.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Velocity reconstruction in the era of DESI and Rubin/LSST. I. Exploring spectroscopic, photometric, and hybrid samples

Peculiar velocities of galaxies and halos can be reconstructed from their spatial distribution alone. This technique is analogous to the baryon acoustic oscillations reconstruction, using the continuity equation to connect density and velocity fields. The resulting reconstructed velocities can be used to measure imprints of galaxy velocities on the cosmic microwave background like the kinematic Sunyaev-Zel’dovich effect or the moving lens effect. As the precision of these measurements increases, characterizing the performance of the velocity reconstruction becomes crucial to allow unbiased and statistically optimal inference. In this paper, we quantify the relevant performance metrics: the variance of the reconstructed velocities and their correlation coefficient with the true velocities. We show that the relevant velocities to reconstruct for kSZ and moving lens are actually the halo—rather than galaxy—velocities. We quantify the impact of redshift-space distortions, photometric redshift errors, satellite galaxy fraction, incorrect cosmological parameter assumptions and smoothing scale on the reconstruction performance. Here, we also investigate hybrid reconstruction methods, where velocities inferred from spectroscopic samples are evaluated at the positions of denser photometric samples. We find that using exclusively the photometric sample is better than performing a hybrid analysis. The 2 Gpc/ℎ length simulations from abacussummit with realistic galaxy samples for DESI and Rubin LSST allow us to perform this analysis in a controlled setting. In the companion paper [B. Hadzhiyska, S. Ferraro, B. Ried Guachalla, and E. Schaan, companion paper, Phys. Rev. D 109, 103534 (2024).], we further include the effects of evolution along the light cone and give realistic performance estimates for DESI luminous red galaxies, emission line galaxies, and Rubin LSST-like samples.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Material Accountancy Performance Indicator Toolkit

The MAPIT - Material Accountancy Performance Indicator Toolkit includes statistical tests and calculations based on open literature that calculate safeguards metrics. The tool can be used with user supplied datasets or with the several included datasets which are generated from models that are maintained at Sandia National Laboratories. The models are not included with the software. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-0467 O

Shoman, Nathan↗

DOE OSTI2024

Assessing equation of state-independent relations for neutron stars with nonparametric models

Relations between neutron star properties that do not depend on the nuclear equation of state offer insights on neutron star physics and have practical applications in data analysis. Such relations are obtained by fitting to a range of phenomenological or nuclear physics equation of state models, each of which may have varying degrees of accuracy. In this study we revisit commonly used relations and reassess them with a very flexible set of phenomenological nonparametric equation of state models that are based on Gaussian processes. Our models correspond to two sets: equations of state which mimic hadronic models, and equations of state with rapidly changing behavior that resemble phase transitions. Here we quantify the accuracy of relations under both sets and discuss their applicability with respect to expected upcoming statistical uncertainties of astrophysical observations. We further propose a goodness-of-fit metric which provides an estimate for the systematic error introduced by using the relation to model a certain equation-of-state set. Overall, the nonparametric distribution is more poorly fit with existing relations, with the I–Love–Q relations retaining the highest degree of universality. Fits degrade for relations involving the tidal deformability, such as the binary-Love and compactness-Love relations, and when introducing phase transition phenomenology. For most relations, systematic errors are comparable to current statistical uncertainties under the nonparametric equation of state distributions.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Metrics to Assess Lightning Context

The Integrated Nuclear Detonation Detection (iNDD) project seeks to develop two statistical constructs. The first will fuse operational data from Nuclear Detonation Detection (NDD) systems in order to determine whether a nuclear event has likely occurred; this is referred to as the “fused detector.” The assorted detection domains of NDD systems span independent regimes with uncorrelated backgrounds; for example, lightning is a background for satellite-based systems, while earthquakes are a background for seismic systems.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Sea Ice Rheology Experiment (SIREx): 1. Scaling and Statistical Properties of Sea-Ice Deformation Fields

As the sea-ice modeling community is shifting to advanced numerical frameworks, developing new sea-ice rheologies, and increasing model spatial resolution, ubiquitous deformation features in the Arctic sea ice are now being resolved by sea-ice models. Initiated at the Forum for Arctic Modeling and Observational Synthesis, the Sea Ice Rheology Experiment (SIREx) aims at evaluating state-of-the-art sea-ice models using existing and new metrics to understand how the simulated deformation fields are affected by different representations of sea-ice physics (rheology) and by model configuration. Part 1 of the SIREx analysis is concerned with evaluation of the statistical distribution and scaling properties of sea-ice deformation fields from 35 different simulations against those from the RADARSAT Geophysical Processor System (RGPS). For the first time, the viscous-plastic (and the elastic-viscous-plastic variant), elastic-anisotropic-plastic, and Maxwell-elasto-brittle rheologies are compared in a single study. We find that both plastic and brittle sea-ice rheologies have the potential to reproduce the observed RGPS deformation statistics, including multi-fractality. Model configuration (e.g., numerical convergence, atmospheric representation, spatial resolution) and physical parameterizations (e.g., ice strength parameters and ice thickness distribution) both have effects as important as the choice of sea-ice rheology on the deformation statistics. It is therefore not straightforward to attribute model performance to a specific rheological framework using current deformation metrics. In light of these results, we further evaluate the statistical properties of simulated Linear Kinematic Features in a SIREx Part 2 companion paper.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Can Simple Metrics Identify the Process(es) Driving Extreme Precipitation?

This work seeks an automatic algorithm to determine the primary meteorological cause(s) of individual extreme precipitation events. Such determinations have been made before, but required a by-hand analysis of each separate event. This is very time-consuming and the field would benefit from an automatic process. This is especially relevant when comparing different datasets to determine which ones most closely hew towards reality. This paper tests three simple metrics over the continental United States using the European Center for Medium-Range Weather Forecasting’s (ECMWF) atmospheric reanalysis (ERA5). The metrics tested measure and compare the strength of three meteorological processes associated with extreme precipitation: fronts, convection, and cyclones. A multivariate statistical technique as well as individual case studies show evidence that the three meteorological processes of interest cannot be isolated from one another using these simple physical metrics. This shows the difficulty in finding “pure” cases of these precipitation-generating processes and suggests approaching these processes with an eye toward mixed-type events.

Swenson, Leif M. (ORCID:0000000199708735)↗

DOE OSTI2020

Developing Metrics to Evaluate the Skill and Credibility of Downscaling

In this project, our overarching goal was to evaluate the skill and credibility of statistical and dynamical downscaling techniques for reconstructing the historical climate record and projecting future climate change. This report summarizes our approach and accomplishments. In the first year of the project, the priority was to generate new data sets. For Year 2, the overall goal was to focus comparisons during the historical period and for a subset of variables, especially temperature and precipitation. In Year 3 and beyond, the focus was mainly on comparisons and metrics for future climate change simulations. To quantify the uncertainty associated with gridded dataset choice, the UCLA group compared eight gridded temperature datasets, examining climatologies, variability, and trends. The Scripps group completed statistical downscaling of NARR with the LOCA method, and UCLA and Scripps collaborated on a comparison of historical and end-of-century temperature and precipitation patterns over California from WRF, LOCA-WRF, and LOCA-Livneh. In addition to downscaling temperature and precipitation, the Scripps group has spent considerable time working on near-surface wind and humidity downscaling. The Santa Clara group assembled the Variable Infiltration Capacity (VIC) for nine snow-dominated river basins draining the western slopes of the Sierra Nevada range and assessed the role of snow albedo feedback in their VIC simulations. The results will help inform future downscaling studies.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Coincident learning for unsupervised anomaly detection of scientific instruments

Abstract Anomaly detection is an important task for complex scientific experiments and other complex systems (e.g. industrial facilities, manufacturing), where failures in a sub-system can lead to lost data, poor performance, or even damage to components. While scientific facilities generate a wealth of data, labeled anomalies may be rare (or even nonexistent), and expensive to acquire. Unsupervised approaches are therefore common and typically search for anomalies either by distance or density of examples in the input feature space (or some associated low-dimensional representation). This paper presents a novel approach called coincident learning for anomaly detection (CoAD), which is specifically designed for multi-modal tasks and identifies anomalies based on coincident behavior across two different slices of the feature space. We define an unsupervised metric, F ^ β , out of analogy to the supervised classification F β statistic. CoAD uses F ^ β to train an anomaly detection algorithm on unlabeled data , based on the expectation that anomalous behavior in one feature slice is coincident with anomalous behavior in the other. The method is illustrated using a synthetic outlier data set and a MNIST-based image data set, and is compared to prior state-of-the-art on two real-world tasks: a metal milling data set and our motivating task of identifying RF station anomalies in a particle accelerator.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Quantitative Particle Analysis of Neptunium-237 Oxides: Optimization of MAMA Analysis for Modified Direct Denitration Products

The production of plutonium-238 through irradiation of neptunium-237 ( 237 Np) target materials for the use in radioisotope thermoelectric generators is paramount for continued deep space exploration. This work employs scanning electron microscopy to analyze 237 Np materials coupled with a well-developed image analysis framework (Morphological Analysis for Material Attribution, or MAMA) to determine the degree of micron-scale homogeneity in the materials. This work demonstrated how the quantification of particle characteristics can validate production materials and affirm the qualitative similarities observed in micrographs. The 237 Np oxide particle analysis determined that the materials from five production runs were quantitatively homogenous (significant at α = 0.05) in particle area, circularity, equivalent circular diameter, and ellipse aspect ratio, with two of the sampling dates having statistically significant different means for one of the four characteristics. Furthermore, these metrics not only confirm general homogeneity of the material but also expand the application of MAMA workflows to 237 Np materials, demonstrating the utility of MAMA analysis for a wider breadth of nuclear materials than previously reported. In the open literature, this study is the first time that these microanalytical techniques were applied to 237 Np materials to this degree.

MAMA↗

DOE OSTI2022

Sizing Considerations for EV Dynamic Wireless Charging Systems with Integrated Energy Storage

Roadways with dynamic wireless charging systems (DWCS) enable charge-sustaining in-motion EV charging, which can reduce charging idle time while increasing range capabilities. Spatially distributed transmitter coils are controlled in response to traffic load that varies significantly minute to minute with high power levels, very short charging time, and low system utilization like wind turbine power. Traffic load estimation and localized analysis may guide effective sizing and topology adoption for feasible and scalable DWCS deployment. DWCS traffic load approximation is reviewed with measured Automated Traffic Recorder (ATR) data and statistical distributions being used to create a synthetic load analyzed using proposed metrics quantifying system utilization over time. Lumped coil section segmentation is compared between second-based distance and spatial density analysis methods, offering 17-27% greater system utilization. A peak load shifting method is proposed for traffic redirection across two tracks with optional BESS integration increasing system utilization by 50-60% depending on time-based and power reserve-based sizing and control.

Lewis, Donovin D.↗