Search NASA⌕ Search

SEARCH · Search NASA

Results for “Statistical Testing Approach”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications

Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq LPUTM accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.

Shanmugavelu, Sanjif↗
DOE OSTI2021

Sub-pixel high-resolution imaging of high-energy x-rays inspired by sub-wavelength optical imaging

We have developed and demonstrated an image super-resolution method—XR-UNLOC: X-Ray UNsupervised particle LOCalization—for hard x-rays measured with fast-frame-rate detectors that is an adaptation of the principle of photo-activated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM), which enabled biological fluorescence imaging at sub-optical-wavelength scales. We demonstrate the approach on experimental coherent Bragg diffraction data measured with 52 keV x-rays from a nanocrystalline sample. From this sample, we resolve the fine fringe detail of a high-energy x-ray Bragg coherent diffraction pattern to an upsampling factor of 16 of the native pixel pitch of 30 μ m of a charge-integrating fastCCD detector. This was accomplished by analysis of individual photon locations in a series of “nearly-dark” instances of the diffraction pattern that each contain only a handful of photons. Central to our approach was the adaptation of the UNLOC photon fitting routine for PALM/STORM to the hard x-ray regime to handle much smaller point spread functions, which required a different statistical test for photon detection and for sub-pixel localization. A comparison to a photon-localization strategy used in the x-ray community (“droplet analysis”) showed that XR-UNLOC provides significant improvement in super-resolution. We also developed a metric by which to estimate the limit of reliable upsampling with XR-UNLOC under a given set of experimental conditions in terms of the signal-to-noise ratio of a photon detection event and the size of the point spread function for guiding future x-ray experiments in many disciplines where detector pixelation limits must be overcome.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRSDate not supplied

QuantifyML: How good is my machine learning model?

This paper presents an approach, QuantifyML, which employs model counting to assess the learnability and robustness of machine learning models. Typically the efficacy of machine learning models is determined by computing their accuracy statistically on test data sets. However, this may be misleading, if the test data is not representative of the problem that is being studied. Further, two different models may have the same accuracy on a given data set, measured statistically, but may be very different in their behavior on unseen data. Also, models with high accuracy could have poor adversarial robustness. In QuantifyML, our goal is to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into a C program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise counts w.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, expressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the robustness of trained models around given inputs. Our evaluation demonstrates these applications of QuantifyML on decision trees and neural networks trained to learn relational properties of graphs, for which we know the ground truth, and to perform image classification, for which we do not have the ground truth, but we can quantify local robustness.

Deep Neural Networks↗
DOE OSTI2019

Fission dynamics of 240Pu from saddle to scission and beyond

Calculations are presented for the time evolution of 240 Pu from the proximity of the outer saddle point until the fission fragments are well separated, using the time-dependent density functional theory extended to superfluid systems. We have tested three families of nuclear energy density functionals and found that all functionals exhibit a similar dynamics: The collective motion is highly dissipative and with little trace of inertial dynamics, due to the one-body dissipation mechanism alone. This finding justifies the validity of using the overdamped collective motion approach and to some extent the main assumptions in statistical models of fission. This conclusion is robust with respect to the nuclear energy density functional used. The configurations and interactions left out of the present theory framework only increase the role of the dissipative couplings. An unexpected finding is varying the pairing strength within a quite large range has only minor effects on the dynamics. We find notable differences in the excitation energy sharing between the fission fragments in the cases of spontaneous and induced fission. With increasing initial excitation energy of the fissioning nucleus, more excitation energy is deposited in the heavy fragment, in agreement with experimental data on average neutron multiplicities.

Bulgac, Aurel↗
DOE OSTI2022

Hypothesis-Agnostic Network-Based Analysis of Real-World Data Suggests Ondansetron is Associated with Lower COVID-19 Any Cause Mortality

Background: The COVID-19 pandemic generated a massive amount of clinical data, which potentially hold yet undiscovered answers related to COVID-19 morbidity, mortality, long-term effects, and therapeutic solutions.Objectives: The objectives of this study were (1) to identify novel predictors of COVID-19 any cause mortality by employing artificial intelligence analytics on real-world data through a hypothesis-agnostic approach and (2) to determine if these effects are maintained after adjusting for potential confounders and to what degree they are moderated by other variables.Methods: A Bayesian statistics-based artificial intelligence data analytics tool (bAIcis®) within the Interrogative Biology® platform was used for Bayesian network learning and hypothesis generation to analyze 16,277 PCR+ patients from a database of 279,281 inpatients and outpatients tested for SARS-CoV-2 infection by antigen, antibody, or PCR methods during the first pandemic year in Central Florida. This approach generated Bayesian networks that enabled unbiased identification of significant predictors of any cause mortality for specific COVID-19 patient populations. These findings were further analyzed by logistic regression, regression by least absolute shrinkage and selection operator, and bootstrapping.Results: We found that in the COVID-19 PCR+ patient cohort, early use of the antiemetic agent ondansetron was associated with decreased any cause mortality 30 days post-PCR+ testing in mechanically ventilated patients.Conclusions: The results demonstrate how a real-world COVID-19-focused data analysis using artificial intelligence can generate unexpected yet valid insights that could possibly support clinical decision making and minimize the future loss of lives and resources.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

SATYA: Defending Against Adversarial Attacks Using Statistical HypothesisTesting

The paper presents a new defense against adversarial attacks for deep neural networks. We demonstrate the effectiveness of our approach against the popular adversarial image generation method DeepFool. Our approach uses Wald’s Sequential Probability Ratio Test to sufficiently sample a carefully chosen neighborhood around an input image to determine the correct label of the image. On a benchmark of 50,000 randomly chosen adversarial images generated by DeepFool we demonstrate that our method SATYA is able to recover the correct labels for 95.76% of the images for CaffeNet and 97.43% of the correct label for GoogLeNet.

Raj, Sunny↗
DOE OSTI2017

Reconciling Basin-Scale Top-Down and Bottom-Up Methane Emission Measurements for Onshore Oil and Gas Development: Cooperative Research and Development Final Report, CRADA Number CRD-14-572

The overall objective of the Research Partnership to Secure Energy for America (RPSEA)-funded research project is to develop independent estimates of methane emissions using top-down and bottom-up measurement approaches and then to compare the estimates, including consideration of uncertainty. Such approaches will be applied at two scales: basin and facility. At facility scale, multiple methods will be used to measure methane emissions of the whole facility (controlled dual tracer and single tracer releases, aircraft-based mass balance and Gaussian back-trajectory), which are considered top-down approaches. The bottom-up approach will sum emissions from identified point sources measured using appropriate source-level measurement techniques (e.g., high-flow meters). At basin scale, the top-down estimate will come from boundary layer airborne measurements upwind and downwind of the basin, using a regional mass balance model plus approaches to separate atmospheric methane emissions attributed to the oil and gas sector. The bottom-up estimate will result from statistical modeling (also known as scaling up) of measurements made at selected facilities, with gaps filled through measurements and other estimates based on other studies. The relative comparison of the bottom-up and top-down estimates made at both scales will help improve understanding of the accuracy of the tested measurement and modeling approaches. The subject of this CRADA is NREL's contribution to the overall project. This project resulted from winning a competitive solicitation no. RPSEA RFP2012UN001, proposal no. 12122-95, which is the basis for the overall project. This Joint Work Statement (JWS) details the contributions of NREL and Colorado School of Mines (CSM) in performance of the CRADA effort.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Laboratory earthquake forecasting: A machine learning competition

Earthquake prediction, the long-sought holy grail of earthquake science, continues to confound Earth scientists. Could we make advances by crowdsourcing, drawing from the vast knowledge and creativity of the machine learning (ML) community? We used Google’s ML competition platform, Kaggle, to engage the worldwide ML community with a competition to develop and improve data analysis approaches on a forecasting problem that uses laboratory earthquake data. The competitors were tasked with predicting the time remaining before the next earthquake of successive laboratory quake events, based on only a small portion of the laboratory seismic data. The more than 4,500 participating teams created and shared more than 400 computer programs in openly accessible notebooks. Complementing the now well-known features of seismic data that map to fault criticality in the laboratory, the winning teams employed unexpected strategies based on rescaling failure times as a fraction of the seismic cycle and comparing input distribution of training and testing data. In addition to yielding scientific insights into fault processes in the laboratory and their relation with the evolution of the statistical properties of the associated seismic data, the competition serves as a pedagogical tool for teaching ML in geophysics. The approach may provide a model for other competitions in geosciences or other domains of study to help engage the ML community on problems of significance.

58 GEOSCIENCES↗
DOE OSTI2025

CMB-S4: Foreground-cleaning Pipeline Comparison for Measuring Primordial Gravitational Waves

We compare multiple foreground-cleaning pipelines for estimating the tensor-to-scalar ratio, r, using simulated maps of the planned CMB-S4 experiment within the context of the South Pole Deep Patch. To evaluate robustness, we analyze bias and uncertainty on r across various foreground suites using map-based simulations. The foreground-cleaning methods include: a parametric maximum likelihood approach applied to auto- and cross-power spectra between frequency maps; a map-based parametric maximum-likelihood method; and a harmonic-space internal linear combination using frequency maps. We summarize the conceptual basis of each method to highlight their similarities and differences. To better probe the impact of foreground residuals, we implement an iterative internal delensing step, leveraging a map-based pipeline to generate a lensing B-mode template from the large aperture telescope frequency maps. Our results show that the performance of the three approaches is comparable for simple and intermediate-complexity foregrounds, with σ(r) ranging from 3–5 ×10 −4 . However, biases at the 1σ–2σ level appear when analyzing more complex forms of foreground emission. By extending the baseline pipelines to marginalize over foreground residuals, we demonstrate that contamination can be reduced to within statistical uncertainties, albeit with a pipeline-dependent impact on σ(r), which translates to a detection significance between 2σ and 4σ for an input value of r = 0.003. These findings suggest varying levels of maturity among the tested pipelines, with the auto- and cross-spectra-based approach demonstrating the best stability and overall performance. Moreover, given the extremely low noise levels, mutual validation of independent foreground-cleaning pipelines is essential to ensure the robustness of any potential detection.

astronomy data analysis↗
NASA NTRS1971

Fatigue failure of materials under narrow band random vibrations. I.

A novel approach for the study of fatigue failure of materials under the multifactor influence of narrow band random vibrations is developed. The approach involves the conduction of an experiment in conjunction with various statistical techniques. Three factors including two statistical properties of the excitation or response are considered and varied simultaneously. A minimum of 6 tests for 3 variables is possible for a fractional f actorial design. The four coefficients of the predicting equation can be independently estimated. A look at 3 predicting equations shows the predominant effect of the root mean square stress of the first order equation.

Huang, T. C.↗
NASA NTRS2011

Method for Identifying Probable Archaeological Sites from Remotely Sensed Data

Archaeological sites are being compromised or destroyed at a catastrophic rate in most regions of the world. The best solution to this problem is for archaeologists to find and study these sites before they are compromised or destroyed. One way to facilitate the necessary rapid, wide area surveys needed to find these archaeological sites is through the generation of maps of probable archaeological sites from remotely sensed data. We describe an approach for identifying probable locations of archaeological sites over a wide area based on detecting subtle anomalies in vegetative cover through a statistically based analysis of remotely sensed data from multiple sources. We further developed this approach under a recent NASA ROSES Space Archaeology Program project. Under this project we refined and elaborated this statistical analysis to compensate for potential slight miss-registrations between the remote sensing data sources and the archaeological site location data. We also explored data quantization approaches (required by the statistical analysis approach), and we identified a superior data quantization approached based on a unique image segmentation approach. In our presentation we will summarize our refined approach and demonstrate the effectiveness of the overall approach with test data from Santa Catalina Island off the southern California coast. Finally, we discuss our future plans for further improving our approach.

Tilton, James C.↗
NASA NTRS2018

Life Qualification of Hall Thrusters By Analysis and Test

High power Hall thrusters under development for deep space robotic and human exploration missions have demanding lifetime and operational requirements that make thruster life qualification challenging. The required burn times make it infeasible to conduct multiple tests with durations exceeding the life requirement, and complex wearout phenomena and the need for power throttling during missions complicate analyses and testing. The traditional qualification approach, a single life test that typically demonstrates 50 to 100% margin on the required lifetime, does not provide enough information to demonstrate low failure risk. It is a weak source of statistical information about the location of the peak in the failure probability distribution and provides no information on the width of the distribution. Testing does provide information that can be used to validate physics-based models of failure processes, however, and the combination can be used to assess mission risk. Validated, conservative, deterministic analysis can be used to demonstrate that most failure processes have such large margins that more detailed analysis is unnecessary. For a subset of the failure modes a more detailed probabilistic analysis is required. Pole erosion in a 12.5 kW Hall thruster is used as a detailed example of this qualification process.

Brophy, John R.↗
DOE OSTI2022

Kinetic theory of particle-in-cell simulation plasma and the ensemble averaging technique

Abstract We derive the kinetic theory of fluctuations in physically and numerically stable particle-in-cell (PIC) simulations of electrostatic plasmas. The starting point is the single-time correlation at the start of the simulation between the statistical fluctuations of the weighted densities of macroparticle centers in the plasma particle phase-space. The fluctuations are associated with different initial conditions, typically due to the random initial conditions (in velocity space) of the macroparticles/simulation plasma, assigned according to their initial distribution of probability. The single-time correlations at all time steps and in each spatial grid cell are then determined from the Laplace–Fourier transforms of the discretized Klimontovich-like equation for the macroparticles and Maxwell’s equations for the fields, as computed by modern PIC codes. We recover the expressions for the electrostatic field and the plasma particle density fluctuation autocorrelation spectra as well as the kinetic equations describing the average evolution of PIC-simulated plasma particles, first derived by Langdon (1970b Proc. 4th Conf. Numerical Simulation of Plasmas ) using a test macroparticle approach perturbing a discretized Vlasovian plasma and then averaging the obtained physical quantity over the initial macroparticle velocity distribution. We generalize and extend these results to the modern algorithms in PIC codes using arbitrary macroparticle weights. Analytical estimates of statistical fluctuation amplitudes are derived as a function of the plasma simulation parameters, using the central limit theorem in the limit of a large number of macroparticles per cell. The theory is then used to analyze the ensemble averaging technique of PIC simulations where statistical averages are performed over ensembles of PIC simulations, modeling the same plasma physics problem but using different statistical realizations of the initial distribution functions of the macroparticles. This method is illustrated by linear Landau damping uncovering (from noise, which is usually considered numerical) the physical fluctuations driven by a single small amplitude electrostatic wave perturbing a PIC simulation plasma in equilibrium.

fluctuations correlations↗
NASA NTRSDate not supplied

Delta Method Application for the Correlation of IR Detector Thermal Parasitic Loads with Statistically Accurate Results

The Linear Etalon Imaging Spectral Array (LEISA) is a cryogenic IR detector that is passively cooled below 110K. LEISA is part of the L’Ralph instrument, which is going on the Lucy mission, with planned flyby’s of the Trojan asteroids around 5.5 AU. One of the primary thermal challenges is to successfully quantify each significant parasitic heat flow value with statistical certainty.Simulation of flight-like thermal environments and correlation of thermal models is an important part of this challenge, which can be critical for thermally sensitive systems such as cryogenic detectors that are passively cooled. It is common practice during TVAC to achieve several thermal balance points and then to correlate several design variables in order to achieve the closest match to all balance points tested. This approach lends itself to high levels of uncertainty in the correlated values due to the cumulative effect of absolute temperature uncertainty and differences between the sensors used for the measurement. Accuracy can be improved with the use of more accurate sensors, by calibrating all sensors with respect to each other, and by using Zero-Q methodologies, all of which add cost and complexity to the test.The Delta Method uses the difference in temperature caused by the changes in the boundary conditions to calculate heat flows and/or thermal resistances across temperature differentials. This eliminates the problem of high uncertainties with absolute temperature measurements, and yields statistically accurate results down to the sensor resolution of 1mK at < 100K, by taking the sensor’s calibration errors out of the equation. During testing, this method relies on parametric variations around an initial balance point, by varying the boundary conditions of the different parasitic heat sources. The Delta Method, as well as its application on the LEISA thermal test, will be discussed in detail.

Daniel G. Bae↗
NASA NTRS1984

The distribution of hot spots

Statistical analyses are compared for two published hot spot data sets, one minimal set of 42 and another larger set of 117, using three different approaches. First, the earths surface is divided into 16 equal-area fractions and the observed distribution of hot spots among them is analyzed using chi-square tests. Second, cumulative distributions about the principal axes of the hot spot inertia tensor are used to describe hot spot distribution. Finally, a hot spot density function is constructed for each of the two hot spot data sets. The methods all indicate that hot spots have a nonuniform distribution, even when statistical fluctuations are considered. To the first order, hot spots are concentrated on one half of of the earth's surface area; within that portion, the distribution is consistent with a uniform distribution. The observed hot spot densities for neither data set are explained solely by plate speed.

Stefanick, M.↗
DOE OSTI2018

A Revised Capsule Design for the Accelerated Testing of Advanced Reactor Fuels

Testing of fast reactor fuel is currently challenging because of the thermal neutron spectrum of the Advanced Test Reactor (ATR) and the fact that ATR coolant is substantially below typical temperatures found in liquid metal-cooled fast reactors. Despite these issues, since 2003 the Idaho National Laboratory (INL) has successfully tested a variety of metallic, oxide, and nitride fuels for fast reactor applications. There are, however, opportunities to improve the experiments so that the time required to reach a desired burnup can be reduced, and so the sensitivity of the experiments to fabrication tolerances can be minimized. A revised capsule design exploits the use of smaller diameter rodlets to increase power densities and correspondingly reduce the irradiation time required to reach high burnup. Thermal and nuclear analyses indicate that the time to reach 30at% burnup could be reduced from about 12 years to approximately 2-3 years if the fuel diameter is reduced by one-half and to around 1-2 years if the diameter is reduced by one-third. In addition, reducing fuel diameter improves the radial power distribution in the fuel without the use of cadmium shrouding. This observation opens up the possibility of testing advanced reactor fuel systems in Small I locations in the ATR where there is substantial unused capacity. Although not directly tied to a revised capsule design, post-irradiation furnace testing is proposed to provide time-to-failure data that will be helpful in the benchmarking of fuel performance models and in assessing the performance of new fuel designs relative to more standard fast reactor fuel systems. In this paradigm, ATR irradiation experiments will be viewed more as preparation for furnace testing than as experiments that exist in isolation. An added benefit to furnace testing is that uncertainties in temperatures will significantly be reduced relative to non-instrumented capsules in the ATR. The time and cost in developing this experimental capability is expected to be reduced considerably because of the availability of the blister anneal furnace used by the HPRR program. A key issue in the experiment approach is the potential for non-prototypical rodlet dimensions to influence the observed performance of a given fuel design. For example, a qualitative assessment indicates that the effects of fuel-cladding chemical interactions during furnace testing might be more pronounced in reduced diameter rodlets simply because thinner cladding would be more sensitive to a given depth of fission product penetration. In a traditional empirically-driven approach to fuel qualification, any change from prototypical behavior is undesirable because the resulting data cannot easily be assimilated into a statistical correlation. However, in the current approach, systematic variations in fuel diameter provide the opportunity to benchmark fuel performance models over a wider range of parameter space, which in turn is expected to improve understanding and result in better, more physically based models in the long term.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

Nonparametric Multiparticle Set Methods for Interpreting Environmental Samples

Collection and analysis of environmental samples is commonly used by a range of stakeholders in nuclear safeguards and security contexts. While the ubiquity of samples and their transport in the environment allow regular collection, developing and demonstrating methods for analyzing these samples is difficult. In this work, an environmental sample consists of a set of one or more individual particles. Recent advances in reactor simulation have allowed us to generate data that are more representative of real-world environmental samples, enabling statistically defensible method development and testing. The most notable of these advances is a drastic increase in the number of material depletion regions, which allows our simulations to capture the variation in isotopic composition seen at length scales consistent with environmental samples. Traditional approaches for handling multiparticle samples treat each particle in the sample individually, estimating the quantity of interest (e.g., core-average burnup) resulting from measurement and analysis of signatures (e.g., nuclide assays) from each individual particle. Individual estimates are then averaged to generate a single estimate of the quantity of interest over the entire sample. In this presentation, we introduce two novel approaches for interpreting environmental samples that comprise of multiple particles: (1) the Quantile-Quantile Comparator, which uses a multivariate generalization of quantile-quantile plots for comparing unknown statistical distributions, and (2) the Set Transformer, an attention-based neural network module designed to model interactions among elements (particles) in the input set (sample). Statistically representative sampling cannot be guaranteed as samples are passively collected and are beholden to what particles are available in the environment. These new analysis methods for set-input problems are expected to be more robust than traditional approaches to issues of sampling bias where particles are not uniformly distributed throughout regions of interest, as well as generally outperform traditional approaches by jointly considering all elements in the set. We will present results comparing the performance of traditional single particle approaches and the novel Quantile-Quantile Comparator and Set Transformer for interpretation of simulated environmental samples.

Phathanapirom, Birdy↗
DOE OSTI2018

Synchrophasor Data Correction under GPS Spoofing Attack: A State Estimation-Based Approach

GPS spoofing attack (GSA) has been shown to be one of the most imminent threats to almost all cyber-physical systems incorporated with the civilian GPS signal. Specifically, for our current agenda of the modernization of the power grid, this may greatly jeopardize the benefits provided by the pervasively installed phasor measurement units (PMU). In this paper, we consider the case where synchrophasor data from PMUs are compromised due to the presence of a single GSA, and show that it can be corrected by signal processing techniques. In particular, we introduce a statistical model for synchrophasorbased power system state estimation (SE), and then derive the spoofing-matched algorithms for synchrophasor data correction against GPS spoofing attack. Different testing scenarios in IEEE 14-, 30-, 57-, 118-bus systems are simulated to show the proposed algorithms’ performance on GSA detection and state estimation. Numerical results demonstrate that our proposed algorithms can consistently locate and correct the spoofed synchrophasor data with good accuracy as long as the system observability is satisfied. The accuracy of state estimation is significantly improved compared with the traditional weighted least square method and approaches the performance under the Genie-aided method.

Phasor measurement units, Global Positioning Syste↗