Search NASA⌕ Search

SEARCH · Search NASA

Results for “neural network Gaussian process”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Analytic Neural Network Gaussian Process Enabled Chance-Constrained Voltage Regulation for Active Distribution Systems with PVs, Batteries and EVs

This paper proposes an analytic neural network Gaussian process (NNGP)-based chance-constrained real-time voltage regulation method for active distribution systems with photovoltaics (PVs), batteries, and electric vehicles (EVs). NNGP can utilize historical measurement data to achieve real-time probabilistic node voltage estimation through Bayesian inference. Then, NNGP is fully analytically embedded into the optimal power flow model to perform voltage regulation and adapt to various topological changes. The uncertainties of voltage estimations are easily considered via the chance constraint, and it has been shown that the adoption of this chance constraint can significantly improve the reliability of voltage regulation under various scenarios. The comparison results with other methods, carried out on a real 759-node distribution system located in western Colorado, U.S., show that the proposed method can achieve accurate voltage estimation across different topologies and reliably perform voltage regulation considering PVs, batteries, and EVs.

active distribution systems↗
DOE OSTI2021

Enhanced Data Efficiency Using Deep Neural Networks and Gaussian Processes for Aerodynamic Design Optimization

Adjoint-based optimization methods are attractive for aerodynamic shape design primarily due to their computational costs being independent of the dimensionality of the input space and their ability to generate high-fidelity gradients that can then be used in a gradient-based optimizer. This makes them very well suited for high-fidelity simulation based aerodynamic shape optimization of highly parametrized geometries such as aircraft wings. However, the development of adjoint-based solvers involve careful mathematical treatment and their implementation require detailed software development. Furthermore, they can become prohibitively expensive when multiple optimization problems are being solved, each requiring multiple restarts to circumvent local optima. In this work, we propose a machine learning enabled, surrogate-based framework that replaces the expensive adjoint solver, without compromising on predicting predictive accuracy. Specifically, we first train a deep neural network (DNN) from training data generated from evaluating the high-fidelity simulation model on a model-agnostic design of experiments on the geometry shape parameters. The optimum shape may then be computed by using a gradient-based optimizer coupled with the trained DNN. Subsequently, we also perform a gradient-free Bayesian optimization, where the trained DNN is used as the prior mean. We observe that the latter framework (DNN-BO) improves upon the DNN-only based optimization strategy for the same computational cost. Overall, this framework predicts the true optimum with very high accuracy, while requiring far fewer high-fidelity function calls compared to the adjoint-based method. Furthermore, we show that multiple optimization problems can be solved with the same machine learning model with high accuracy, to amortize the offline costs associated with constructing our models. Our methodology finds applications in the early stages of aerospace design. (C) 2021 Published by Elsevier Masson SAS.

Renganathan, S. Ashwin↗
DOE OSTI2020

Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels

While deep neural networks (DNNs) and Gaussian Processes (GPs) are both popularly utilized to solve problems in reinforcement learning, both approaches feature undesirable drawbacks for challenging problems. DNNs learn complex non-linear embeddings, but do not naturally quantify uncertainty and are often data-inefficient to train. GPs infer posterior distributions over functions, but popular kernels exhibit limited expressivity on complex and high-dimensional data. Fortunately, recently discovered conjugate and neural tangent kernel functions encode the behavior of overparameterized neural networks in the kernel domain. We demonstrate that these kernels can be efficiently applied to regression and reinforcement learning problems by analyzing a baseline case study.We apply GPs with neural network dual kernels to solve reinforcement learning tasks for the first time. We demonstrate, using the well understood mountain-car problem, that GPs empowered with dual kernels perform at least as well as those using the conventional radial basis function kernel. Finally, we conjecture that by inheriting the probabilistic rigor of GPs and the powerful embedding properties of DNNs, GPs using NN dual kernels will empower future reinforcement learning models on difficult domains.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Multi-fidelity modeling to predict the rheological properties of a suspension of fibers using neural networks and Gaussian processes

Unveiling the rheological properties of fiber suspensions is of paramount interest to many industrial applications. There are multiple factors, such as fiber aspect ratio and volume fraction, that play a significant role in altering the rheological behavior of suspensions. Three-dimensional (3D) numerical simulations of coupled differential equations of the suspension of fibers are computationally expensive and time-consuming. Machine learning algorithms can be trained on the available data and make predictions for the cases where no numerical data are available. However, some widely used machine learning surrogates, such as neural networks, require a relatively large training dataset to produce accurate predictions. Multi-fidelity models, which combine high-fidelity data from numerical simulations and less expensive lower fidelity data from resources such as simplified constitutive equations, can pave the way for more accurate predictions. Here, we focus on neural networks and the Gaussian processes with two levels of fidelity, i.e., high and low fidelity networks, to predict the steady-state rheological properties, and compare them to the single-fidelity network. High-fidelity data are obtained from direct numerical simulations based on an immersed boundary method to couple the fluid and solid motion. The low-fidelity data are produced by using constitutive equations. Multiple neural networks and the Gaussian process structures are used for the hyperparameter tuning purpose. Results indicate that with the best choice of hyperparameters, both the multi-fidelity Gaussian processes and neural networks are capable of making predictions with a high level of accuracy with neural networks demonstrating marginally better performance.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

A comparison of Gaussian processes and neural networks for computer model emulation and calibration

The Department of Energy relies on complex physics simulations for prediction in domains like cosmology, nuclear theory, and materials science. These simulations are often extremely computationally intensive, with some requiring days or weeks for a single simulation. In order to assure their accuracy, these models are calibrated against observational data in order to estimate inputs and systematic biases. Because of their great computational complexity, this process typically requires the construction of an emulator, a fast approximation to the simulation. In this paper, two emulator approaches are compared: Gaussian process regression and neural networks. Their emulation accuracy and calibration performance on three real problems of Department of Energy interest is considered. On these problems, the Gaussian process emulator tends to be more accurate with narrower, but still well-calibrated uncertainty estimates. The neural network emulator is accurate, but tends to have large uncertainty on its predictions. Finally, as a result, calibration with the Gaussian process emulator produces more constrained posteriors that still perform well in prediction.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Boron Coordination in Multicomponent Glasses: Analytical Models and Machine Learning With Uncertainty

Borosilicate glasses are extensively used in a variety of applications from kitchenware to nuclear waste immobilization due to the strong network formed by the Si-O-B bond that makes it resistant to chemical corrosion and gives it a low thermal expansion. Boron, however, exists in both trigonal BO3 and tetrahedral BO4 bonds in glass systems, which impacts the chemical durability and thermal resistance of the glass, amongst other properties. Boron coordination (N4), or the ratio of the amount of BO4 to BO3 within a glass, may aid in predicting these properties but is difficult to derive without experimental data due to the complexity of impacts from varied glass compositions and processing factors. For this reason, compositional models have been developed to predict boron coordination, but the models typically include a limited number of glass components. To help fill this gap in the models, in this work, a diverse multicomponent glass dataset of 809 glasses is compiled from a literature search, and then a number of analytical and machine learning (ML) models are trained on the dataset. Previously developed modified Bernstein and modified Du Stebbins analytical models were fitted to update parameters with the new dataset. Then, partially Bayesian neural networks, Gaussian process regressor, and heteroskedastic deterministic neural networks were evaluated. The ML models examined all have different strategies to overcome the potential for overfitting as a result of a limited training dataset, and return results that account for model uncertainty, which can be valuable for understanding model reliability. For the first time, cooling rate is introduced as an input parameter for ML models, showing consistent improvements in performance and solidifying the importance of including parameters outside of composition alone for N4 prediction. The machine learning models examined here show promise in accurate predictions of boron coordination in borosilicate glasses, all achieving R2 values of 0.91.

boron coordination↗
DOE OSTI2022

Accelerating cosmological inference with Gaussian processes and neural networks – an application to LSST Y1 weak lensing and galaxy clustering

ABSTRACT Studying the impact of systematic effects, optimizing survey strategies, assessing tensions between different probes and exploring synergies of different data sets require a large number of simulated likelihood analyses, each of which cost thousands of CPU hours. In this paper, we present a method to accelerate cosmological inference using emulators based on Gaussian process regression and neural networks. We iteratively acquire training samples in regions of high posterior probability which enables accurate emulation of data vectors even in high dimensional parameter spaces. We showcase the performance of our emulator with a simulated 3×2 point analysis of LSST-Y1 with realistic theoretical and systematics modelling. We show that our emulator leads to high-fidelity posterior contours, with an order of magnitude speed-up. Most importantly, the trained emulator can be re-used for extremely fast impact and optimization studies. We demonstrate this feature by studying baryonic physics effects in LSST-Y1 3×2 point analyses where each one of our MCMC runs takes approximately 5 min. This technique enables future cosmological analyses to map out the science return as a function of analysis choices and survey strategy.

Astronomy & Astrophysics↗
DOE OSTI2021

Correspondence of NNGP Kernel and the Matérn Kernel

Kernels representing limiting cases of neural network architectures have recently gained popularity. However, the application and performance of these new kernels compared to existing options, such as the Matérn kernel, is not well studied. We take a practical approach to explore the neural network Gaussian process (NNGP) kernel and its application to data in Gaussian process regression. We first demonstrate the necessity of normalization to produce valid NNGP kernels and explore related numerical challenges. We further demonstrate that the predictions from this model are quite inflexible, and therefore do not vary much over the valid hyperparameter sets. We then demonstrate a surprising result that the predictions given from the NNGP kernel correspond closely to those given by the Matérn kernel under specific circumstances, which suggests a deep similarity between overparameterized deep neural networks and the Matérn kernel. Finally, we demonstrate the performance of the NNGP kernel as compared to the Matérn kernel on three benchmark data cases, and we conclude that for its flexibility and practical performance, the Matérn kernel is preferred to the novel NNGP in practical applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Uncertainty Quantification for Data-Driven Machine Learning Models in Nuclear Engineering Applications: Where We Are and What Do We Need?

Machine learning (ML) has been leveraged to tackle a diverse range of tasks in almost all branches of nuclear engineering. Many of the successes in ML applications can be attributed to the recent performance breakthroughs in deep learning, the growing availability of computational power, data, and easy-to-use ML libraries. However, these empirical successes have often outpaced our formal understanding of the ML algorithms. An important but under-rated area is uncertainty quantification (UQ) of ML. ML-based models are subject to approximation uncertainty when they are used to make predictions, due to sources including but not limited to, data noise, data coverage, extrapolation, imperfect model architecture and the stochastic training process. The goal of this paper is to clearly explain and illustrate the importance of UQ of ML. We will elucidate the differences in the basic concepts of UQ of physics-based models and data-driven ML models. Various sources of uncertainties in physical modeling and data-driven modeling will be discussed, demonstrated, and compared. We will also present and demonstrate a few techniques to quantify the ML prediction uncertainties, including Monte Carlo dropout, deep ensemble, Bayesian neural networks, Gaussian Processes and conformal prediction. Lastly, we will discuss the need for building a verification, validation and UQ framework to establish ML credibility.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Task-oriented machine learning surrogates for tipping points of agent-based models

We present a machine learning framework bridging manifold learning, neural networks, Gaussian processes, and Equation-Free multiscale approach, for the construction of different types of effective reduced order models from detailed agent-based simulators and the systematic multiscale numerical analysis of their emergent dynamics. The specific tasks of interest here include the detection of tipping points, and the uncertainty quantification of rare events near them. Our illustrative examples are an event-driven, stochastic financial market model describing the mimetic behavior of traders, and a compartmental stochastic epidemic model on an Erdös-Rényi network. We contrast the pros and cons of the different types of surrogate models and the effort involved in learning them. Importantly, the proposed framework reveals that, around the tipping points, the emergent dynamics of both benchmark examples can be effectively described by a one-dimensional stochastic differential equation, thus revealing the intrinsic dimensionality of the normal form of the specific type of the tipping point. This allows a significant reduction in the computational cost of the tasks of interest.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

MLMOD: Machine Learning Methods for Data-Driven Modeling in LAMMPS

MLMOD is a software package for incorporating machine learning approaches and models into simulations of microscale mechanics and molecular dynamics in LAMMPS. Recent machine learning approaches provide promising data-driven approaches for learning representations for system behaviors from experimental data and high fidelity simulations. The package facilitates learning and using data-driven models for (i) dynamics of the system at larger spatial-temporal scales (ii) interactions between system components, (iii) features yielding coarser degrees of freedom, and (iv) features for new quantities of interest characterizing system behaviors. MLMOD provides hooks in LAMMPS for (i) modeling dynamics and time-step integration, (ii) modeling interactions, and (iii) computing quantities of interest characterizing system states. The package allows for use of machine learning methods with general model classes including Neural Networks, Gaussian Process Regression, Kernel Models, and other approaches. Here we discuss our prototype C++/Python package, aims, and example usage. For related papers, examples, updates, and additional information see https://github.com/atzberg/mlmod and http://atzberger.org/.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Wide neural networks with bottlenecks are deep Gaussian processes

There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width. The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP, finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through extreme post-bottleneck depths, and prevents the kernel of the network from losing discriminative power at extreme post-bottleneck depths.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

43 PARTICLE ACCELERATORS↗
DOE OSTI2020

Genetic Algorithm for Hyperparameter Optimization in Gaussian Process Modeling

A genetic algorithm is developed and applied to optimize hyperparameters of convolutional recursively determined dual neural network-Gaussian process (NNGP) kernels. As a specific application of the combined GPNN-GA algorithm, it is applied to image classification in publicly available data of Hyper Suprime-Cam Subaru Strategic Program. Matthews correlation coefficient is calculated based on results of binary star-galaxy classification and used as a fitting function of the GA module of the algorithm. The simulation results confirm significant improvement of the classification accuracy with optimized hyperparameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Reduced Order Models Generation for HTGRs Pebble Shuffling Procedure Optimization Studies

This report provides an initial study for producing reduced-order models (ROMs) of pebble-bed high temperature gas reactor (HTGR) models for the purposes of design optimization. As an initial study, this work is meant to be exploratory---identifying useful workflows and methods for ROM generation---and not meant to be a catch-all analysis of HTGR ROM generation and usage for optimization. This report summarizes three tasks performed in Fiscal Year 2022: 1) the creation of HTGR model, 2) the sensitivity analysis of model design parameters, and 3) an introduction to ROM generation techniques. The representative HTGR model created in this work is a multiphysics equilibrium-core using the BlueCRAB (comprehensive reactor analysis bundle) reactor analysis application, coupling four physical phenomena: neutronics, streamline depletion, porous flow thermal hydraulics, and pebble heat conduction. Part of the model creation was identifying some design parameters and quantities of interest that are relevant in an optimization analysis and adjustable in the model. The sensitivity analysis utilized a polynomial chaos expansion methodology to compute global sensitivity metrics. This analysis showed that thermal hydraulics parameters and quantities of interest had a relatively small impact on simulation results. Finally, the ROM generation work involved exploring three different ROM methodologies: polynomial regression, a Gaussian process, and artificial neural networks. Using a cross-validation technique to characterize ROM performance, the Gaussian process and single-layer artificial neural networks showed the most promising results. Overall, this study was insightful and the lessons learned will be invaluable for the eventual development of an HTGR design optimization workflow.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial

On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. In conclusion, two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).

97 MATHEMATICS AND COMPUTING↗