Search NASA⌕ Search

SEARCH · Search NASA

Results for “Statistical Testing Approach”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

A Statistical Testing Approach for Quantifying Software Reliability; Application to an Example System

The U.S. Nuclear Regulatory Commission (NRC) encourages the use of probabilistic risk assessment (PRA) technology in all regulatory matters, to the extent supported by the state-of-the-art in PRA methods and data. Although much has been accomplished in the area of risk-informed regulation, risk assessment for digital systems has not been fully developed. The NRC established a plan for research on digital systems to identify and develop methods, analytical tools, and regulatory guidance for (1) including models of digital systems in the PRAs of nuclear power plants (NPPs), and (2) incorporating digital systems in the NRC's risk-informed licensing and oversight activities. Under NRC's sponsorship, Brookhaven National Laboratory (BNL) explored approaches for addressing the failures of digital instrumentation and control (I and C) systems in the current NPP PRA framework. Specific areas investigated included PRA modeling digital hardware, development of a philosophical basis for defining software failure, and identification of desirable attributes of quantitative software reliability methods. Based on the earlier research, statistical testing is considered a promising method for quantifying software reliability. This paper describes a statistical software testing approach for quantifying software reliability and applies it to the loop-operating control system (LOCS) of an experimental loop of the Advanced Test Reactor (ATR) at Idaho National Laboratory (INL).

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2020

Statistical Modeling and Analysis of k -Layer Coverage of Two-Dimensional Materials in Inkjet Printing Processes

Two-dimensional layered materials/flakes, also known as crystalline atom-thick layer nanosheets, have recently been receiving great attention in electronics fabrication due to their unique and intriguing properties. The k-layer coverage area (i.e., the area covered by k number of overlapping layers) of the printed flake pattern significantly impacts on the properties of the printed electronics. In this work, we constructed a statistical model to describe the k-layer coverage of randomly distributed two-dimensional materials. A series of results are obtained to provide not only the expectation but also the variance of the coverage area. The boundary effects on the random flakes coverage are also studied. In addition, an approximated statistical testing approach is also developed in this work to detect abnormal coverage patterns. Furthermore, the case studies based on simulated data and real flakes images obtained from the inkjet printing process demonstrate the accuracy and effectiveness of the proposed model and analysis methods.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Causal Interactions between Southern Ocean Polynyas and High-Latitude Atmosphere–Ocean Variability

Abstract Weddell Sea open-ocean polynyas have been observed to occasionally release heat from the deep ocean to the atmosphere, indicating that their sporadic appearances may be an important feature of high-latitude atmosphere–ocean variability. Yet, observations of the phenomenon are sparse and many standard-resolution models represent these features poorly, if at all. We use a fully coupled, synoptic-scale preindustrial control simulation of the Energy Exascale Earth System Model (E3SMv0-HR) to effectively simulate open-ocean polynyas and investigate their role in the climate system. Our approach employs statistical tests of Granger causality to diagnose local and remote drivers of, and responses to, polynya heat loss on interannual to decadal time scales. First, we find that polynya heat loss Granger causes a persistent increase in surface air temperature over the Weddell Sea, strengthening the local cyclonic wind circulation. Along with responding to polynyas, atmospheric conditions also facilitate their development. When the Southern Ocean experiences a rapid poleward shift in the circumpolar westerlies following a prolonged negative phase of the southern annular mode (SAM), Weddell Sea salinity increases, promoting density destratification and convection in the water column. Finally, we find that the reduction of surface heat fluxes during periods of full ice cover is not fully compensated by ocean heat transport into the high latitudes. This imbalance leads to a buildup of ocean heat content that supplies polynya heat loss. These results disentangle the complex, coupled climate processes that both enable the polynya’s existence and respond to it, providing insights to improve the representation of these highly episodic sea ice features in climate models.

Kaufman, Zachary S.↗
DOE OSTI2025

Testing convolutional neural network based deep learning systems: a statistical metamorphic approach

Machine learning technology spans many areas and today plays a significant role in addressing a wide range of problems in critical domains,i.e., healthcare, autonomous driving, finance, manufacturing, cybersecurity,etc. Metamorphic testing (MT) is considered a simple but very powerful approach in testing such computationally complex systems for which either an oracle is not available or is available but difficult to apply. Conventional metamorphic testing techniques have certain limitations in verifying deep learning-based models (i.e., convolutional neural networks (CNNs)) that have a stochastic nature (because of randomly initializing the network weights) in their training. In this article, we attempt to address this problem by using a statistical metamorphic testing (SMT) technique that does not require software testers to worry about fixing the random seeds (to get deterministic results) to verify the metamorphic relations (MRs). We propose seven MRs combined with different statistical methods to statistically verify whether the program under test adheres to the relation(s) specified in the MR(s). We further use mutation testing techniques to show the usefulness of the proposed approach in the healthcare space and test two CNN-based deep learning models (used for pneumonia detection among patients). The empirical results show that our proposed approach uncovers 85.71% of the implementation faults in the classifiers under test (CUT). Furthermore, we also propose an MRs minimization algorithm for the CUT, thus saving computational costs and organizational testing resources.

Computer Science↗
DOE OSTI2026

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗
DOE OSTI2024

Predictive Modeling of NOx Emissions from Lean Direct Injection of Hydrogen and Hydrogen/Natural Gas Blends Using Flame Imaging and Machine Learning

This research paper explores the use of machine learning to relate images of flame structure and luminosity to measured NOx emissions. Images of reactions produced by 16 aero-engine derived injectors for a ground-based turbine operated on a range of fuel compositions, air pressure drops, preheat temperatures and adiabatic flame temperatures were captured and postprocessed. The experimental investigations were conducted under atmospheric conditions, capturing CO, NO and NOx emissions data and OH* chemiluminescence images from 27 test conditions. The injector geometry and test conditions were based on a statistically designed test plan. These results were first analyzed using the traditional analysis approach of analysis of variance (ANOVA). The statistically based test plan yielded 432 data points, leading to a correlation for NOx emissions as a function of injector geometry, test conditions and imaging responses, with 70.2% accuracy. As an alternative approach to predicting emissions using imaging diagnostics as well as injector geometry and test conditions, a random forest machine learning algorithm was also applied to the data and was able to achieve an accuracy of 82.6%. This study offers insights into the factors influencing emissions in ground-based turbines while emphasizing the potential of machine learning algorithms in constructing predictive models for complex systems.

08 HYDROGEN↗
DOE OSTI2018

Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Thermodynamics of order and randomness in dopant distributions inferred from atomically resolved imaging

Abstract Exploration of structure-property relationships as a function of dopant concentration is commonly based on mean field theories for solid solutions. However, such theories that work well for semiconductors tend to fail in materials with strong correlations, either in electronic behavior or chemical segregation. In these cases, the details of atomic arrangements are generally not explored and analyzed. The knowledge of the generative physics and chemistry of the material can obviate this problem, since defect configuration libraries as stochastic representation of atomic level structures can be generated, or parameters of mesoscopic thermodynamic models can be derived. To obtain such information for improved predictions, we use data from atomically resolved microscopic images that visualize complex structural correlations within the system and translate them into statistical mechanical models of structure formation. Given the significant uncertainties about the microscopic aspects of the material’s processing history along with the limited number of available images, we combine model optimization techniques with the principles of statistical hypothesis testing. We demonstrate the approach on data from a series of atomically-resolved scanning transmission electron microscopy images of Mo x Re 1- x S 2 at varying ratios of Mo/Re stoichiometries, for which we propose an effective interaction model that is then used to generate atomic configurations and make testable predictions at a range of concentrations and formation temperatures.

25 ENERGY STORAGE↗
DOE OSTI2020

XRD-PUAT (X-Ray Diffraction - Parameter Uncertainty Analysis Toolkit)

This XRD uncertainty toolkit was built to investigate uncertainty and local least squares topology over a specified parameter space on a single histogram in a gpx GSAS-II file. The parameter uncertainty can be investigated using either a frequentist F-test approach or a Bayesian Inference statistical inversion method on a weighted least squares or peak fit refinements.SAND Number: SAND2020-12227 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Moore, Alexander↗
DOE OSTI2018

Bootstrap inversion technique for atmospheric trace gas source detection and quantification using long open-path laser measurements

Advances in natural gas extraction technology have led to increased activity in the production and transport sectors in the United States and, as a consequence, an increased need for reliable monitoring of methane leaks to the atmosphere. We present a statistical methodology in combination with an observing system for the detection and attribution of fugitive emissions of methane from distributed potential source location landscapes such as natural gas production sites. We measure long (> 500 m), integrated open-path concentrations of atmospheric methane using a dual frequency comb spectrometer and combine measurements with an atmospheric transport model to infer leak locations and strengths using a novel statistical method, the non-zero minimum bootstrap (NZMB). The new statistical method allows us to determine whether the empirical distribution of possible source strengths for a given location excludes zero. Using this information, we identify leaking source locations (i.e., natural gas wells) through rejection of the null hypothesis that the source is not leaking. The method is tested with a series of synthetic data inversions with varying measurement density and varying levels of model–data mismatch. It is also tested with field observations of (1) a non-leaking source location and (2) a source location where a controlled emission of 3.1 × 10 -5 kg s -1 of methane gas is released over a period of several hours. This series of synthetic data tests and outdoor field observations using a controlled methane release demonstrates the viability of the approach for the detection and sizing of very small leaks of methane across large distances (4 + km 2 in synthetic tests). The field tests demonstrate the ability to attribute small atmospheric enhancements of 17 ppb to the emitting source location against a background of combined atmospheric (e.g., background methane variability) and measurement uncertainty of 5 ppb (1 σ ), when measurements are averaged over 2 min. The results of the synthetic and field data testing show that the new observing system and statistical approach greatly decreases the incidence of false alarms (that is, wrongly identifying a well site to be leaking) compared with the same tests that do not use the NZMB approach and therefore offers increased leak detection and sizing capabilities.

03 NATURAL GAS↗
DOE OSTI2017

A statistical approach to combining multisource information in one-class classifiers

A new method is introduced in this paper for combining information from multiple sources to support one-class classification. The contributing sources may represent measurements taken by different sensors of the same physical entity, repeated measurements by a single sensor, or numerous features computed from a single measured image or signal. The approach utilizes the theory of statistical hypothesis testing, and applies Fisher's technique for combining p-values, modified to handle nonindependent sources. Classifier outputs take the form of fused p-values, which may be used to gauge the consistency of unknown entities with one or more class hypotheses. The approach enables rigorous assessment of classification uncertainties, and allows for traceability of classifier decisions back to the constituent sources, both of which are important for high-consequence decision support. Application of the technique is illustrated in two challenge problems, one for skin segmentation and the other for terrain labeling. Finally, the method is seen to be particularly effective for relatively small training samples.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Shared micromobility as a first- and last-mile transit solution? Spatiotemporal insights from a novel dataset

The first- and last-mile (FM/LM) problem is a major deterrent to public transit use. With the rise of shared micromobility options such as shared e-scooters in recent years, there is a growing interest in understanding their potential to serve as a last-mile transit solution. However, empirical data regarding the integrated use of shared micromobility and public transit have been limited so far. As a result, much is unknown regarding the spatiotemporal patterns and characteristics of shared micromobility trips serving as an FM/LM connection to transit. Here, this paper addresses these knowledge gaps by leveraging a novel dataset (i.e., the Spin post-ride survey dataset) that records thousands of transit-connecting shared e-scooter trips in Washington DC. Specifically, we used the dataset to reveal the spatiotemporal patterns of transit-connecting shared e-scooter trips in Washington DC, resulting in some major policy insights regarding the integral use of shared e-scooters and public transit. We further leveraged the dataset to validate if and to what extent a commonly applied buffer-zone approach can infer FM/LM micromobility trips accurately. Statistical tests showed that the actual FM/LM Spin e-scooter trips differ from inferred FM/LM Spin e-scooter trips in both spatial and temporal dimensions. This indicates that the common practice of inferring FM/LM micromobility trips with a buffer-zone approach can lead to inaccurate estimates of transit-connecting micromobility trips.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

Testing the impact of electromagnetic fields on the directed flow of constituent quarks in heavy-ion collisions

It has been proposed that strong electromagnetic fields produced in the early stages of heavy-ion collisions can lead to splitting of the rapidity-odd directed flow of positive and negative hadrons. For light hadrons, the interpretation of such measurements is complicated by the low magnitude of directed flow as well as by ambiguities arising from transported quarks. To overcome these complications, we propose measurements using only hadrons carrying produced quarks (u¯, d¯, s s¯ ). Here, we discuss how to identify the kinematics where such hadrons are produced via the coalescence mechanism and therefore their flow is the sum of the flow of their constituent quarks. With this sum rule verified for certain combinations of hadrons, the expected systematic violation of this rule with increasing electric charge can be measured, which could be a consequence of the electromagnetic fields produced in the collisions. Our approach can be tested with the high statistics data from Phase II of the Beam Energy Scan program at the Relativistic Heavy Ion Collider.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

Challenges of Using Indistinguishable from Background as a Release Criterion at FUSRAP Projects - 18157

The US Army Corps of Engineers (USACE) utilizes the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM) approach to site release on Formerly Utilized Sites Remedial Action Program (FUSRAP) projects. Use of Indistinguishable from Background (IFB) as a release criterion is discussed in MARSSIM as Scenario B; however, the discussion is arguably limited and refers the user to other documents. USACE has utilized the IFB approach at select FUSRAP projects. USACE experience is that use of IFB as a release criterion has been difficult to implement at best. There are many challenges in using the IFB approach. These include: release criterion is required per MARSSIM; multiple reference areas need to be established; and data presentation. Potential methods to overcome these challenges are discussed. USACE has utilized alternative statistical approaches as well as the IFB approach discussed in MARSSIM for specific purposes at FUSRAP sites. Alternatives to the IFB approach discussed in this paper include other statistical tests such as the Tukey Kramer test. These alternatives may be better suited to release an area of a property prior to the development of release criteria. USACE has found using the IFB criterion useful in certain specific circumstances. These include post MARSSIM Scenario A final status survey data evaluations and for review of historical data. The challenges, alternatives, and circumstances to use the IFB approach are discussed. Other environmental remediation projects attempting to utilize IFB as a criterion may benefit from the USACE experiences as discussed herein. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2017

Evaluation of PCR Systems for Field Screening of Bacillus anthracis

There is little published data on the performance of hand-portable polymerase chain reaction (PCR) instruments that could be used by first responders to determine if a suspicious powder contains a potential biothreat agent. We evaluated five commercially available hand-portable PCR instruments for detection of Bacillus anthracis (Ba). We designed a cost-effective, statistically-based test plan that allows instruments to be evaluated at performance levels ranging from 0.85-0.95 lower confidence bound (LCB) on the probability of detection (POD) at confidence levels of 80-95%. We assessed specificity using purified genomic DNA from 13 Ba strains and 18 Bacillus near neighbors, interference with 22 common hoax powders encountered in the field, and PCR inhibition when Ba spores were spiked into these powders. Our results indicated that three of the five instruments achieved >0.95 LCB on the POD with 95% confidence at test concentrations of 2,000 genome equivalents/mL (comparable to 2,000 spores/mL), displaying more than sufficient sensitivity for screening suspicious powders. These instruments exhibited no false positive results or PCR inhibition with common hoax powders, and reliably detected Ba spores spiked into common hoax powders, though some issues with instrument controls were observed. Our testing approach enables efficient instrument performance testing to a statistically rigorous and cost-effective test plan to generate performance data that will allow users to make informed decisions regarding the purchase and use of biodetection equipment in the field.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2018

Understanding Dynamical Systems in High-Dimensional Parameter Spaces (Final Report)

In the last few decades, computational models, especially of the complex dynamical systems of greatest interest, i.e., climate, high energy density science, combustion, etc. have become increasingly accurate and sophisticated. Consequently, they are now routinely used to predict outcomes of experiments or to inform design decisions. However, many such simulations depend on a (large) number of parameters in order to, for example, approximate physical phenom whose exact solution is unknown or describe material properties in situations with little or no experimental data. Choosing these parameters and understanding how the outcomes of such a simulation changes with respect to the inputs are rapidly becoming some of the primary challenges in many application areas. Simulations have long been too complex for humans to fully comprehend and thus are often treated as black-box systems. The inputs to the systems are typically a collection of parameters with expert-selected ranges, and the outputs are various quantities of interest (QoIs) describing the performance of the system. Therefore, the fundamental challenge can be described as one of understanding a high-dimensional (many input parameters), multi-variate (many QoIs) function. Unfortunately, existing analysis tools are only of limited use for this task as most techniques can neither deal with many input dimensions nor offer opportunities to jointly study several QoIs. Furthermore, data analysis has long relied on an exploratory process in which scientist first visualize the available data, for example, using scatter plots or 3D renderings, form new hypotheses on some phenomena of interest, and subsequently test these hypotheses using dedicated analysis or statistics approaches. However, for high-dimensional functions this exploration is difficult if not impossible using existing tools as there are too many QoIs to simultaneously observe, and visualizations of high-dimensional spaces are abstract and difficult to understand if they exist at all. The goal of this project is to build new analysis techniques and tool that offer high level insights for exploring high-dimensional scientific data. And to extend existing approach beyond single scalar functions and instead consider multiple QoIs simultaneously.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Material Control & Accounting Modeling Developments for a Generic TRISO Fuel Fabrication Facility

The Material Protection, Accounting, and Control Technologies (MPACT) program utilizes modeling and simulation to assess Material Control and Accountability (MC&A) concerns for a variety of nuclear facilities. The Sandia National Laboratories (SNL)-developed Fissile Facility Flow Modeler (F3M) and the Material Accountancy Performance Indicator Toolkit (MAPIT) have historically provided MPACT with the capability to analyze MC&A approaches for nuclear facilities to determine that these facilities meet regulatory requirements. In FY25, improvements on the application of the F3M and MAPIT tools to simulate a generic TRi-structural ISOtropic (TRISO) fuel fabrication facility were successfully completed. The generic TRISO fuel fabrication F3M model captures the entire TRISO fuel fabrication process and is adaptable to any final TRISO fuel form, including spherical pebbles and cylindrical compacts loaded into graphite prismatic blocks. Comprehensive F3M/MAPIT functionality for the generic TRISO fuel fabrication model has been demonstrated. This modeling framework can be applied to support the U.S. Department of Energy and domestic nuclear industry stakeholders in developing MC&A approaches for advanced fuel fabrication facilities via statistical tests that demonstrate compliance to regulatory requirements.

97 MATHEMATICS AND COMPUTING↗