Search NASA⌕ Search

SEARCH · Search NASA

Results for “Bayesian Neural Network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Online MCMC Thinning with Kernelized Stein Discrepancy

A fundamental challenge in Bayesian inference is efficient representation of a target distribution. Many nonparametric approaches do so by sampling a large number of points using variants of Markov chain Monte Carlo (MCMC). Here, we propose an MCMC variant that retains only those posterior samples which exceed a kernelized Stein discrepancy (KSD) threshold, which we call KSD thinning. We establish the convergence and complexity trade-offs for several settings of KSD thinning as a function of the KSD threshold parameter, sample size, and other problem parameters. We provide experimental comparisons against other online nonparametric Bayesian methods that generate low-complexity posterior representations. We observe superior consistency/complexity trade-offs across a range of settings including MCMC sampling on two Bayesian inference problems from the biological sciences, and 10 × inference speedup and storage reduction for Bayesian neural networks with no loss of accuracy and no increase in training time. Our code is available at https://github.com/colehawkins/KSD-Thinning.

Bayesian inference↗

Neural network ensembles and uncertainty estimation for predictions of inelastic mechanical deformation using a finite element method-neural network approach

The finite element method (FEM) is widely used to simulate a variety of physics phenomena. Approaches that integrate FEM with neural networks (NNs) are typically leveraged as an alternative to conducting expensive FEM simulations in order to reduce the computational cost without significantly sacrificing accuracy. However, these methods can produce biased predictions that deviate from those obtained with FEM, since these hybrid FEM-NN approaches rely on approximations trained using physically relevant quantities. In this work, an uncertainty estimation framework is introduced that leverages ensembles of Bayesian neural networks to produce diverse sets of predictions using a hybrid FEM-NN approach that approximates internal forces on a deforming solid body. The uncertainty estimator developed herein reliably infers upper bounds of bias/variance in the predictions for a wide range of interpolation and extrapolation cases using a three-element FEM-NN model of a bar undergoing plastic deformation. This proposed framework offers a powerful tool for assessing the reliability of physics-based surrogate models by establishing uncertainty estimates for predictions spanning a wide range of possible load cases.

42 ENGINEERING↗

Advanced Semi-Supervised Learning with Uncertainty Estimation for Phase Identification in Distribution Systems

The integration of advanced metering infrastructure (AMI) into power distribution networks generates valuable data for tasks such as phase identification; however, the limited and unreliable availability of labeled data in the form of customer phase connectivity presents challenges. To address this issue, we propose a semi-supervised learning (SSL) framework that effectively leverages labeled and unlabeled data. Our approach incorporates self-training, label spreading, and Bayesian neural networks (BNNs) to enhance phase identification with AMI data. Our method uses an ensemble of multilayer perceptron classifiers in a self-training setup, iteratively adding high-confidence pseudo-labels to improve robustness. We also apply label spread to propagate labels based on data similarity, which enhances generalization across diverse distributions. In addition, we employ a BNNs with uncertainty estimation, boosting confidence in predictions and reducing phase identification errors. In our case study, we achieved approximately 98% +/- 0.08 accuracy with uncertainty using minimal and unreliable labeled data from a real U.S. utility, Duquesne Light Company. Our SSL approach, combined with uncertainty estimation, provides an efficient solution for phase identification in AMI data, ultimately improving the reliability of smart grid applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Advanced Semi-Supervised Learning With Uncertainty Estimation for Phase Identification in Distribution Systems

The integration of advanced metering infrastructure (AMI) into power distribution networks generates valuable data for tasks such as phase identification; however, the limited and unreliable availability of labeled data in the form of customer phase connectivity presents challenges. To address this issue, we propose a semi-supervised learning (SSL) framework that effectively leverages labeled and unlabeled data. Our approach incorporates self-training, label spreading, and Bayesian neural networks (BNNs) to enhance phase identification with AMI data. Our method uses an ensemble of multilayer perceptron classifiers in a self-training setup, iteratively adding high-confidence pseudo-labels to improve robustness. We also apply label spread to propagate labels based on data similarity, which enhances generalization across diverse distributions. In addition, we employ a BNNs with uncertainty estimation, boosting confidence in predictions and reducing phase identification errors. In our case study, we achieved approximately 98% +/- 0.08 accuracy with uncertainty using minimal and unreliable labeled data from a real U.S. utility, Duquesne Light Company. Our SSL approach, combined with uncertainty estimation, provides an efficient solution for phase identification in AMI data, ultimately improving the reliability of smart grid applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Variation-Resilient FeFET-Based In-Memory Computing Leveraging Probabilistic Deep Learning

Reliability issues stemming from device level nonidealities of nonvolatile emerging technologies like ferroelectric field-effect transistors (FeFETs), especially at scaled dimensions, cause substantial degradation in the accuracy of in-memory crossbar-based AI systems. Here, in this work, we present a variation-aware design technique to characterize the device level variations and to mitigate their impact on hardware accuracy employing a Bayesian neural network (BNN) approach. An effective conductance variation model is derived from the experimental measurements of cycle-to-cycle (C2C) and device-to-device (D2D) variations performed on FeFET devices fabricated using 28 nm high-k metal gate technology. The variations were found to be a function of different conductance states within the given programming range, which sharply contrasts earlier efforts where a fixed variation dispersion was considered for all conductance values. Such variation characteristics formulated for three different device sizes at different read voltages were provided as prior variation information to the BNN to yield a more exact and reliable inference. Near-ideal accuracy for shallow networks (MLP5 and LeNet models) on the MNIST dataset and limited accuracy decline by ~3.8%–16.1% for deeper AlexNet models on CIFAR10 dataset under a wide range of variations corresponding to different device sizes and read voltages, demonstrates the efficacy of our proposed device-algorithm co-design technique.

97 MATHEMATICS AND COMPUTING↗

Uncertainty Quantification for Data-Driven Machine Learning Models in Nuclear Engineering Applications: Where We Are and What Do We Need?

Machine learning (ML) has been leveraged to tackle a diverse range of tasks in almost all branches of nuclear engineering. Many of the successes in ML applications can be attributed to the recent performance breakthroughs in deep learning, the growing availability of computational power, data, and easy-to-use ML libraries. However, these empirical successes have often outpaced our formal understanding of the ML algorithms. An important but under-rated area is uncertainty quantification (UQ) of ML. ML-based models are subject to approximation uncertainty when they are used to make predictions, due to sources including but not limited to, data noise, data coverage, extrapolation, imperfect model architecture and the stochastic training process. The goal of this paper is to clearly explain and illustrate the importance of UQ of ML. We will elucidate the differences in the basic concepts of UQ of physics-based models and data-driven ML models. Various sources of uncertainties in physical modeling and data-driven modeling will be discussed, demonstrated, and compared. We will also present and demonstrate a few techniques to quantify the ML prediction uncertainties, including Monte Carlo dropout, deep ensemble, Bayesian neural networks, Gaussian Processes and conformal prediction. Lastly, we will discuss the need for building a verification, validation and UQ framework to establish ML credibility.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A bi-level advanced control framework for large-scale control of buildings with system-level impact

Increased electricity consumption combined with new forms of generation is testing the reliability of our grid infrastructure. This work describes a method to improve the reliability of the grid through large-scale advanced building control. This paper develops a bi-level distributed control framework to shift the load of 153 buildings to achieve a system-level objective of tracking a power reference signal. This bi-level control is based on the previously-developed ANPV-MPC, a predictive controller that uses a Bayesian neural network to generate an accurate control model and adapt to changing conditions over time. By shifting the building electricity demand to better match the available power, the grid system supplying the buildings is more reliable as evidenced by the analysis of node voltages across an IEEE 13-bus distribution system. The proposed bi-level control framework tracks the system-level power reference with enough accuracy to regulate node voltages across the IEEE 13-bus distribution system within ANSI limits of ±5%. Additionally, the adaptive nature of ANPV-MPC allows each building across the system to adapt to changing conditions, further amplifying the system-level reliability.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

TensorBNN: Bayesian inference for neural networks using TensorFlow

We report that TensorBNN is a new package based on TensorFlow that implements Bayesian inference for modern neural network models. The posterior density of neural network model parameters is represented as a point cloud sampled using Hamiltonian Monte Carlo. The TensorBNN package leverages TensorFlow's architecture and its ability to use modern graphics processing units in both the training and prediction stages.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Nuclear masses learned from a probabilistic neural network

Machine learning methods and uncertainty quantification have been gaining interest throughout the last several years in low-energy nuclear physics. In particular, Gaussian processes and Bayesian neural networks have increasingly been applied to improve mass model predictions while providing well-quantified uncertainties. In this work, we use the probabilistic Mixture Density Network (MDN) to directly predict the mass excess of the 2016 Atomic Mass Evaluation within the range of measured data, and we extrapolate the inferred models beyond available experimental data. The MDN provides not only mean values but also full posterior distributions both within the training set and extrapolated testing set. We show that the addition of physical information to the feature space increases the accuracy of the match to the training data as well as provides for more physically meaningful extrapolations beyond the the limits of experimental data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Electronic structure prediction of medium and high entropy alloys across composition space

We propose machine learning (ML) models to predict the electron density — the fundamental unknown of a material’s ground state — across the composition space of concentrated alloys. From this, other physical properties can be inferred, enabling accelerated exploration. A significant challenge is that the number of descriptors and sampled compositions required for accurate prediction grows rapidly with species. To address this, we employ Bayesian Active Learning (AL), which minimizes training data requirements by leveraging uncertainty quantification capabilities of Bayesian Neural Networks. Compared to the strategic tessellation of the composition space, Bayesian-AL reduces the number of training data points by a factor of 2.5 for ternary (SiGeSn) and 1.7 for quaternary (CrFeCoNi) systems. We also introduce easy-to-optimize, body-attached-frame descriptors, which respect physical symmetries while keeping descriptor-vector size nearly constant as alloy complexity increases. Our ML models demonstrate high accuracy and generalizability in predicting both electron density and energy across composition space.

materials science↗

Quantification of neural networks uncertainties with applications to SAFARI-1 axial neutron flux profiles

Deep Neural Networks (DNNs) have been widely used as a data-driven modelling tool in nuclear engineering. However, as a Machine Learning model, Artificial Neural Network (ANN) predictions are subjected to uncertainties originating from the noise in training data, incomplete coverage of the domain, and imperfect neural network architectures. In this work, we target at quantifying the prediction/approximation uncertainties of ANNs using Monte Carlo Dropout (MCD), as well as Bayesian Neural Networks (BNNs) which are solved by variational inference. With a demonstration problem in which neural networks are used to predict the assembly axial neutron flux profiles, the results have shown that the three different neural network models (regular DNNs, DNNs solved with MCD and BNNs) can produce results that agree very well among each other and with the measurement data, on cycles that are not used in the training process. Besides the excellent generalization capability, the uncertainty bands produced by MCD and BNN agree very well, and in general, they can fully envelop the noisy measurement data points. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Precision calibration of calorimeter signals in the ATLAS experiment using an uncertainty-aware neural network

The ATLAS experiment at the Large Hadron Collider explores the use of modern neural networks for a multi-dimensional calibration of its calorimeter signal defined by clusters of topologically connected cells (topo-clusters). The Bayesian neural network (BNN) approach not only yields a continuous and smooth calibration function that improves performance relative to the standard calibration but also provides uncertainties on the calibrated energies for each topo-cluster. The results obtained by using a trained BNN are compared to the standard local hadronic calibration and to a calibration provided by training a deep neural network. The uncertainties predicted by the BNN are interpreted in the context of a fractional contribution to the systematic uncertainties of the trained calibration. They are also compared to uncertainty predictions obtained from an alternative estimator employing repulsive ensembles.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Improving Predictions Under Uncertainty of Material Plasma Device Operations

Understanding the properties of materials when exposed to various plasma temperatures and fluxes is essential to the building and operating of fusion reactors. The Material Plasma Exposure eXperiment (MPEX) is an instrument currently being developed by the Department of Energy (DOE) for this purpose. MPEX is expected to come online in stages over the next five years. Proto-MPEX, the predecessor to MPEX, operated from 2014 to 2021, and was designed to understand the generation of plasma temperatures and fluxes at orders of magnitude below what will be obtained by MPEX. This work uses the recently developed stochastic neural network (SNN), a machine learning technique capable of operating under uncertainty to provide a surrogate model for the Proto-MPEX device. We demonstrate that SNN outperforms Bayesian neural network (BNN), a standard in the field of machine learning with uncertainty. The development of a robust surrogate of the Proto-MPEX will aid in the commissioning and operation of the MPEX device.

Archibald, Rick↗

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing

In Hezaveh et al. (2017) we showed that deep learning can be used for model parameter estimation and trained convolutional neural networks to determine the parameters of strong gravitational lensing systems. Here we demonstrate a method for obtaining the uncertainties of these parameters. We review the framework of variational inference to obtain approximate posteriors of Bayesian neural networks and apply it to a network trained to estimate the parameters of the Singular Isothermal Ellipsoid plus external shear and total flux magnification. We show that the method can capture the uncertainties due to different levels of noise in the input data, as well as training and architecture-related errors made by the network. To evaluate the accuracy of the resulting uncertainties, we calculate the coverage probabilities of marginalized distributions for each lensing parameter. By tuning a single hyperparameter, the dropout rate, we obtain coverage probabilities approximately equal to the confidence levels for which they were calculated, resulting in accurate and precise uncertainty estimates. Our results suggest that neural networks can be a fast alternative to Monte Carlo Markov Chains for parameter uncertainty estimation in many practical applications, allowing more than seven orders of magnitude improvement in speed.

Perreault Levasseur, Laurence↗

Modelling stellar activity with Gaussian process regression networks

ABSTRACT Stellar photospheric activity is known to limit the detection and characterization of extrasolar planets. In particular, the study of Earth-like planets around Sun-like stars requires data analysis methods that can accurately model the stellar activity phenomena affecting radial velocity (RV) measurements. Gaussian Process Regression Networks (GPRNs) offer a principled approach to the analysis of simultaneous time series, combining the structural properties of Bayesian neural networks with the non-parametric flexibility of Gaussian Processes. Using HARPS-N solar spectroscopic observations encompassing three years, we demonstrate that this framework is capable of jointly modelling RV data and traditional stellar activity indicators. Although we consider only the simplest GPRN configuration, we are able to describe the behaviour of solar RV data at least as accurately as previously published methods. We confirm the correlation between the RV and stellar activity time series reaches a maximum at separations of a few days, and find evidence of non-stationary behaviour in the time series, associated with an approaching solar activity minimum.

Camacho, J. D. (ORCID:0000000151215560)↗

A Framework for the Analysis of Deep Neural Networks in Autonomous Aerospace Applications using Bayesian Statistics

Deep Neural Networks (DNNs) are considered to be key components in many autonomous systems. Applications range from vision-based obstacle avoidance to intelligent/learning control and planning. Safety-critical applications as found in the aerospace domain require that the behavior of the DNN is validated and tested rigorously for safety of the autonomous system (AUS). In this paper, we present a framework to support testing of DNNs and the analysis of the network structure. Our framework employs techniques from statistical modeling and active learning to effectively generate test cases for DNN safety testing and performance analysis. We will present results of a case study on a physics-based Deep recurrent residual neural network (DR-RNN), which has been trained to emulate the aerodynamics behavior of a fixed-wing aircraft.

Deep Neural networks↗

Machine Learning Assisted Safety Modeling and Analysis of Advanced Reactors

With the advances in computational power and numerical methods, analysts can now rely on first-principle simulations to predict ultra-fine details in a variety of applications. Advances in machine learning (ML) have produced algorithms that can now learn high-level abstractions via hierarchical models. This project aims to leverage advances in ML techniques and the available high-resolution simulation data to develop a novel modeling and simulation (M\&S) methodology for reactor safety analysis. While application-agnostic ML techniques are available, complex physics constraints need to be incorporated into ML techniques to build ML-based closures for computationally efficient predictive simulations. This project intends to develop a physics-guided data-driven multi-scale methodology for M\&S of advanced reactors. The project focuses on thermal fluid (T/F) phenomena, which play major roles in advanced reactor safety. Specifically, we propose a data-driven coarse-mesh turbulence model based on local flow features for the transient analysis of thermal mixing and stratification in a sodium-cooled fast reactor (SFR). The model has a coarse-mesh setup to ensure computational efficiency, while it is trained by fine-mesh computational fluid dynamics (CFD) data with Reynolds-averaged Navier-Stokes (RANS) turbulence model to ensure accuracy. Three different neural networks are developed and tested for loss-of-flow transients in the hot pool of SFR, i.e. the densely connected convolutional neural network (DCNN), long-short-term-memory network based on proper orthogonal decomposition (POD-LSTM), and the DCNN informed by LSTM (DCNN-LSTM). The performances of these three neural networks are evaluated based on baseline models. The DCNN-LSTM model has been chosen for further hyperparameter optimization. Furthermore, based on a simplified two-dimensional case, uncertainty quantification (UQ) of the developed ML-based closure are investigated with three methods, i.e. Monte Carlo dropout, deep ensemble, and Bayesian neural network. The developed ML-based turbulent viscosity closure relation based on deep ensemble is then integrated into the system analysis module SAM and serves as a term in the conservation equations. Such a SAM-ML based procedure guarantees that the obtained results are consistent with the physical constraints of the thermal-fluid system. The SAM-ML simulation on the same loss-of-flow transient showed comparable accuracy with the CFD simulation but with a much coarser mesh setup. Last but not least, the ML-based closure improvement with the support of higher-fidelity data from large eddy simulation (LES) is discussed. As a first step towards this direction, a baseline LES simulation is performed to obtain comparable data with RANS results. Based on the early results, future investigation on further improving the ML-based closure is discussed. We believe the developed approach that combines scientific machine learning with nuclear system analysis code can benefit the advanced reactor community as more accurate safety analyses will better characterize reactor safety margins and reduce licensing efforts.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial

On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. In conclusion, two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).

97 MATHEMATICS AND COMPUTING↗