Search NASA⌕ Search

SEARCH · Search NASA

Results for “neural network training”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Training Spiking Neural Networks Using Combined Learning Approaches

Spiking neural networks (SNNs), the class of neural networks used in neuromorphic computing, are difficult to train using traditional back-propagation techniques. Spike timingdependent plasticity (STDP) is a biologically inspired learning mechanism that can be used to train SNNs. Evolutionary algorithms have also been demonstrated as a method for training SNNs. In this work, we explore the relationship between these two training methodologies. We evaluate STDP and evolutionary optimization as standalone methods for training networks, and also evaluate a combined approach where STDP weight updates are applied within an evolutionary algorithm. We also apply Bayesian hyperparameter optimization as a meta learner for each of the algorithms. We find that STDP by itself is not an ideal learning rule for randomly connected networks, while the inclusion of STDP within an evolutionary algorithm leads to similar performance, with a few interesting differences. This study suggests future work in understanding the relationship between network topology and learning rules.

Elbrecht, Daniel↗
DOE OSTI2023

An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems

We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, in stead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves tremendous amount of time spent on training neural networks. Another essential aspect of EnSF is its analytical update step, gradually incorporating data information into the score function, which is crucial in mitigating the degeneracy issue faced when dealing with very high-dimensional nonlinear filtering problems. High-dimensional Lorenz systems are used to demonstrate the performance of our method. EnSF provides surprisingly impressive performance in reliably tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimension) with highly nonlinear observation processes, which is a well-known challenging problem for existing filtering methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Deep Learning At Depth: Estimating subsurface parameters from geophysical monitoring data

Geophysical imaging techniques are a non-invasive way to image the subsurface and understand both subsurface solid (rock/soil) and fluid property distributions and their evolution in time. Inversions of the geophysical data, such as Electrical Resistance Tomography (ERT) data, are solved to estimate the subsurface property distributions, such as conductivity, and many inversion techniques smooth out sharp gradients in rock or fluid property distributions. Sharp gradients in subsurface properties tend to be present in situations with complex subsurface structures, which are common in many subsurface applications. We have successfully demonstrated that it is possible to inform, or constrain, inversions with neural networks trained on synthetic data with complex subsurface structures. Initial results suggest this process may be optimizable to yield property distributions that better represent the true property distributions than the same inversion process without the neural network constraint. Future work would optimize the neural network performance for this application and then apply the synthetic-data trained neural network to real data to understand the utility and performance of this technique for real data sets.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Search for $t\bar{t}H/A \rightarrow t\bar{t}t\bar{t}$ production in proton–proton collisions at $\sqrt{s}=13$ $\text {TeV}$ with the ATLAS detector

A search is presented for a heavy scalar (H) or pseudo-scalar (A) predicted by the two-Higgs-doublet models, where the H/A is produced in association with a top-quark pair $(t\bar{t}H/A),$ and with the H/A decaying into a $t\bar{t}$ pair. The full LHC Run 2 proton–proton collision data collected by the ATLAS experiment is used, corresponding to an integrated luminosity of $139~\text {fb}^{-1}.$ Events are selected requiring exactly one or two opposite-charge electrons or muons. Data-driven corrections are applied to improve the modelling of the $t\bar{t}$ +jets background in the regime with high jet and b-jet multiplicities. These include a novel multi-dimensional kinematic reweighting based on a neural network trained using data and simulations. An H/A-mass parameterised graph neural network is trained to optimise the signal-to-background discrimination. In combination with the previous search performed by the ATLAS Collaboration in the multilepton final state, the observed upper limits on the $t\bar{t}H/A \rightarrow t\bar{t}t\bar{t}$ production cross-section at 95% confidence level range between 14 fb and 5.0 fb for an H/A with mass between 400 GeV and 1000 GeV , respectively. Assuming that both the H and A contribute to the $t\bar{t}t\bar{t}$ cross-section, tan β values below 1.7 or 0.7 are excluded for a mass of 400 GeV or 1000 GeV , respectively. The results are also used to constrain a model predicting the pair production of a colour-octet scalar, with the scalar decaying into a $t\bar{t}$ pair.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

SuperCDMS Event Reconstruction Using Convolutional Neural Networks [Thesis]

The SuperCDMS experiment uses cryogenic silicon and germanium detectors to search for dark matter candidates such as WIMPs (Weakly Interacting Massive Particles) streaming through the Earth. Collisions in the silicon and germanium crystals are expected to produce phonons whose thermal signatures can be measured. This thesis first describes the integration of a new Signal Distribution Unit (SDU) to the SuperCDMS data acquisition system, which allows for synchronization of multiple detectors and electronic/mechanical noise characterization via accelerometer, antenna, and AC phase measurements. From SuperCDMS detector data it is necessary to reconstruct the energies of the particle events. This thesis explores the use of Convolutional Neural Networks (CNNs) to perform this reconstruction and finds that, although they perform well, changing the noise model breaks the model and requires the neural network to be retrained. In order to mitigate this issue, a new CNN model is proposed which includes the noise Power Spectral Density (PSD) of the data as an additional input to the CNN. While it proves to be effective as a denoising algorithm, it still fails for data with a different noise model. However, including data from multiple PSDs in the neural network training sample allows it to handle data with different types of noise while still maintaining the quality of the reconstruction. Nevertheless, neural networks trained even on multiple PSDs do not robustly handle data taken with PSDs dissimilar to those in the training sample, suggesting that CNNs may need to be retrained whenever the noise environment changes in a significant way.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Machine learning and atomic layer deposition: Predicting saturation times from reactor growth profiles using artificial neural networks

In this work, we explore the application of deep neural networks to the optimization of atomic layer deposition (ALD) processes. In particular, we focus on a one-shot optimization problem, where we try to predict the optimal dose time that leads to saturation everywhere in the reactor based on thickness values measured at different points of an ALD reactor after a single trial growth. In order to tackle this problem, we introduce a dataset designed to train neural networks to predict saturation times based on these inputs for a cross-flow ALD reactor. Here, we then explore the predictive ability of artificial neural networks of different depths and sizes using a separate testing dataset to evaluate their accuracies. The results obtained show that networks trained using stochastic gradient descent methods can accurately predict saturation times without requiring any additional information on the surface kinetics. This provides a viable approach to minimize the number of experiments required to optimize new ALD processes in a known reactor, and it highlights the way machine learning can be leveraged for thin film growth and manufacturing. While the datasets and training procedure depend on the reactor geometry, the trained neural networks provide a general surrogate model connecting thickness values and trial dose times with optimal saturation times that can be reused for different ALD processes within the same reactor.

36 MATERIALS SCIENCE↗
DOE OSTI2022

deep-taxon v0.1

deep-taxon is a package for training neural networks for classifying genomic sequence data. deep-taxon provides tools for converting genomic sequence data into high performance computing file formats capable of training neural networks at scale.

Tritt, Andrew↗
DOE OSTI2023

Quantum Neural Networks: Issues, Training, and Applications

Our work in the field aims at explaining the limitations and expressive power of Quantum Machine Learning models, as well as finding feasible training algorithms that could be implemented in near-term Quantum Computers. The promise of Quantum Machine Learning is that by incorporating quantum effects, such as entanglement, into machine learning models researchers can improve model performance and understand more complex datasets. This pledge is particularly pronounced in the design of Quantum neural networks (QNNs), a promising framework for creating quantum algorithms, that promise to outperform classical models by combining the speedups of quantum computation with the widespread successes of deep learning. We show that applying this approach alone to quantum deep learning is problematic given that an excess of entanglement between the hidden and visible layers can destroy the predictive power of our QNN models. We address the barren plateau problem by suggesting the use of a generative, unbounded, nonlinear loss function with simple gradients. The loss function quantifies how much the quantum states generated by the QNNs differ from the data and the goal during training is to minimize it. Finally, we showcase how to use generative training to construct a "classical-quantum" neural network to accurately interpolate between the ground states of a Molecular Hamiltonian, a central question in Quantum Chemistry.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Neural Architecture and Feature Search for Predicting the Ridership of Public Transportation Routes

Accurately predicting the ridership of public-transit routes provides substantial benefits to both transit agencies, who can dispatch additional vehicles proactively before the vehicles that serve a route become crowded, and to passengers, who can avoid crowded vehicles based on publicly available predictions. The spread of the coronavirus disease has further elevated the importance of ridership prediction as crowded vehicles now present not only an inconvenience but also a public-health risk. At the same time, accurately predicting ridership has become more challenging due to evolving ridership patterns, which may make all data except for the most recent records stale. One promising approach for improving prediction accuracy is to fine-tune the hyper-parameters of machine-learning models for each transit route based on the characteristics of the particular route, such as the number of records. However, manually designing a machine-learning model for each route is a labor-intensive process, which may require experts to spend a significant amount of their valuable time. To help experts with designing machine-learning models, we propose a neural-architecture and feature search approach, which optimizes the architecture and features of a deep neural network for predicting the ridership of a public-transit route. Our approach is based on a randomized local hyper-parameter search, which minimizes both prediction error as well as the complexity of the model. We evaluate our approach on real-world ridership data provided by the public transit agency of Chattanooga, TN, and we demonstrate that training neural networks whose architectures and features are optimized for each route provides significantly better performance than training neural networks whose architectures and features are generic.

Ayman, Afiya↗
DOE OSTI2021

Condensed Phase Water Molecular Multipole Moments from Deep Neural Network Models Trained on Ab Initio Simulation Data

Ionic solvation phenomena in liquids involve intense interactions in the inner solvation shell. For interactions beyond the first shell, the ion–solvent interaction energies result from the sum of many smaller-magnitude contributions that can still include polarization effects. Deep neural network (DNN) methods have recently found wide application in developing efficient molecular models that maintain near-quantum accuracy. In this work, we extend the DeePMD-kit code to produce accurate molecular multipole moments in the bulk and near interfaces. The new method is validated by comparing the DNN moments with those generated by ab initio simulations. The moments are used to compute the electrostatic potential at the center of a molecular-sized hydrophobic cavity in water. The results show that the fields produced by the DNN models are in quantitative agreement with the AIMD-derived values. These efficient methods will open the door to more accurate solvation models for large solutes such as proteins.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

A systematic approach to generating accurate neural network potentials: the case of carbon

Abstract Availability of affordable and widely applicable interatomic potentials is the key needed to unlock the riches of modern materials modeling. Artificial neural network-based approaches for generating potentials are promising; however, neural network training requires large amounts of data, sampled adequately from an often unknown potential energy surface. Here we propose a self-consistent approach that is based on crystal structure prediction formalism and is guided by unsupervised data analysis, to construct an accurate, inexpensive, and transferable artificial neural network potential. Using this approach, we construct an interatomic potential for carbon and demonstrate its ability to reproduce first principles results on elastic and vibrational properties for diamond, graphite, and graphene, as well as energy ordering and structural properties of a wide range of crystalline and amorphous phases.

Chemistry↗
DOE OSTI2021

Adversarial Perturbations Are Not So Weird: Entanglement of Robust and Non-Robust Features in Neural Network Classifiers

Neural networks trained on visual data are well-known to be vulnerable to often imperceptible adversarial perturbations. The reasons for this vulnerability are still being debated in the literature. Recently Ilyas et al. (2019) showed that this vulnerability arises, in part, because neural network classifiers rely on highly predictive but brittle “non-robust” features. In this paper we extend the work of Ilyas et al. by investigating the nature of the input patterns that give rise to these features. In particular, we hypothesize that in a neural network trained in a standard way, non-robust features respond to small, “non-semantic” patterns that are typically entangled with larger, robust patterns, known to be more human-interpretable, as opposed to solely responding to statistical artifacts in a dataset. Thus, adversarial examples can be formed via minimal perturbations to these small, entangled patterns. In addition, we demonstrate a corollary of our hypothesis: robust classifiers are more effective than standard (non-robust) ones as a source for generating transferable adversarial examples in both the untargeted and targeted settings. The results we present in this paper provide new insight into the nature of the non-robust features responsible for adversarial vulnerability of neural network classifiers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Autonomous nondestructive evaluation of resistance spot welded joints

The application of non-destructive evaluation approaches has attracted strong interests in modern automotive industries. Here, we present an autonomous deep-computing framework to analyze raw videos from infrared systems and to predict weld nugget shape and size with unprecedented accuracy and speed. In a comprehensive training and testing experiment with 90 videos (seven sets of welding material stack-ups), a new method was developed to assemble sufficient datasets for neural network training. Our framework successfully predicts all the nugget shapes with F1 scores that range from 0.84 to 0.92. The total training time on Nvidia DGX station takes less than 10 min for each set of welding material stack-up. The real inference time of an individual dataset (with 30 video frames) takes about 0.005 s. The procedure and methods developed in the study can be applied to other image-based weld property prediction, as well as other manufacturing processes. Furthermore, our well-trained neural networks take limited memory resources (2.3 MB) and are suitable for embedded microprocessors for in-situ welding quality control as edge computing within an intelligent welding framework.

42 ENGINEERING↗
DOE OSTI2020

Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks, Frontiers in Earth Science: Dataset

This data package contains watershed modeling inputs and outputs as well as deep neural networks training and testing results used in "Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks" (Cromwell et al., 2021). We train various deep neural network models with different architectures to predict subsurface permeability from stream discharge hydrograph at the watershed outlet. The training data are obtained from ensemble simulations of hydrographs corresponding to an permeability ensemble using a fully-distributed, integrated surface-subsurface hydrologic model. The trained model is then applied to estimate the permeability of the real watershed using its observed hydrograph at the outlet. Our study demonstrates that the permeabilities of the soil and geologic facies that make significant contributions to the outlet discharge can be more accurately estimated from the discharge data. Their estimations are also more robust with observation errors. Compared to the traditional ensemble smoother method, DNNs show stronger performance in capturing the nonlinear relationship between permeability and stream hydrograph to accurately estimate permeability. Our study sheds new light on the value of the emerging deep learning methods in assisting integrated watershed modeling by improving parameter estimation, which will eventually reduce the uncertainty in predictive watershed models. For detailed information regarding watershed model description and DNNs setup, please refer to Cromwell et al., 2021. The deep_learning.zip file contains the inputs and outputs for the different deep neural networks. The watershed_model.zip file contains the inputs and outputs for watershed simulation using ATS. The figures.zip file contains raw figures and their corresponding scripts. For a more detailed description, please see the README.md file within each compressed file.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Improving ADAM through an implicit-explicit (IMEX) time-stepping approach

The ADAM optimizer, often used in machine learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. Here, this work shows that the classical ADAM algorithm is a first-order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions of the ADAM scheme obtained by using higher-order IMEX methods to solve the ODE. Based on this approach, we derive a new optimization algorithm for neural network training that performs better than classical ADAM on several regression and classification problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Neural network-based classification and regression of magnetohydrodynamic modes in tokamaks

We present a machine learning-based magnetohydrodynamic (MHD) classifier and regressor that utilizes real or complex-valued 3D magnetic sensor array data to determine neoclassical tearing mode (NTM) onset times in tokamaks with millisecond accuracy. The input dataset consists of poloidal profiles of complex Fourier amplitudes with an n = 1 toroidal mode number from 144 human-labeled ITER Baseline Scenario discharges in the DIII-D tokamak, spanning both tearing-dominated and sawtooth-dominated regimes. Since m, n = 2,1 NTMs frequently emerge alongside sawteeth at the same frequency in this scenario, the focus is on isolating the m = 1 and m = 2 components of the n = 1 MHD mode near the tearing onset. To improve model regularization and prediction stability, singular value decomposition was applied to balance the sawtooth and tearing datasets. The enriched datasets facilitated training neural networks that learn the key distinguishing features of sawtooth and tearing modes in the poloidal profiles of their magnetic amplitude and phase. When the modes occur independently, the networks achieve perfect classification due to the modes’ distinct characteristics and low measurement noise. In the more experimentally relevant case where both modes coexist, the networks maintain exceptional performance across key metrics. Tests on synthetic data with known ground truth demonstrate the superior accuracy of the neural network trained on complex-valued input compared to models using real amplitude, phase, or pseudo-complex data, achieving both a mean time delay and standard deviation below 1 ms. Notably, standard linear regression methods fitting the dominant singular modes to the data closely match the neural network’s performance. Applying these methods across a broad range of H-mode scenarios will enable future studies to systematically identify dominant NTM triggers as scenario-specific variables, paving the way for more effective tearing mode avoidance strategies in future fusion reactor designs.

machine learning↗
DOE OSTI2022

Evaluating cloud liquid detection against Cloudnet using cloud radar Doppler spectra in a pre-trained artificial neural network

Detection of liquid-containing cloud layers in thick mixed-phase clouds or multi-layer cloud situations from ground-based remote-sensing instruments still poses observational challenges, yet improvements are crucial since the existence of multi-layer liquid layers in mixed-phase cloud situations influences cloud radiative effects, cloud lifetime, and precipitation formation processes. Hydrometeor target classifications such as from Cloudnet that require a lidar signal for the classification of liquid are limited to the maximum height of lidar signal penetration and thus often lead to underestimations of liquid-containing cloud layers. Here we evaluate the Cloudnet liquid detection against the approach of Luke et al. (2010) which extracts morphological features in cloud-penetrating cloud radar Doppler spectra measurements in an artificial neural network (ANN) approach to classify liquid beyond full lidar signal attenuation based on the simulation of the two lidar parameters particle backscatter coefficient and particle depolarization ratio. We show that the ANN of Luke et al. (2010) which was trained under Arctic conditions can successfully be applied to observations at the mid-latitudes obtained during the 7-week-long ACCEPT field experiment in Cabauw, the Netherlands, in 2014. In a sensitivity study covering the whole duration of the ACCEPT campaign, different liquid-detection thresholds for ANN-predicted lidar variables are applied and evaluated against the Cloudnet target classification. Independent validation of the liquid mask from the standard Cloudnet target classification against the ANN-based technique is realized by comparisons to observations of microwave radiometer liquid-water path, ceilometer liquid-layer base altitude, and radiosonde relative humidity. In addition, a case-study comparison against the cloud feature mask detected by the space-borne lidar aboard the CALIPSO satellite is presented. Three conclusions were drawn from the investigation. First, it was found that the threshold selection criteria of liquid-related lidar backscatter and depolarization alone control the liquid detection considerably. Second, all threshold values used in the ANN framework were found to outperform the Cloudnet target classification for deep or multi-layer cloud situations where the lidar signal is fully attenuated within low liquid layers and the cloud radar is able to detect the microphysical fingerprint of liquid in higher cloud layers. Third, if lidar data are available, Cloudnet is at least as good as the ANN. The times when Cloudnet outperforms the ANN in liquid detections are often associated with situations where cloud dynamics smear the imprint of cloud microphysics on the radar Doppler spectra.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection

Deep neural networks (DNNs) have achieved state-of-the-art performance across a variety of traditional machine learning tasks, e.g., speech recognition, image classification, and segmentation. The ability of DNNs to efficiently approximate high-dimensional functions has also motivated their use in scientific applications, e.g., to solve partial differential equations and to generate surrogate models. In this paper, we consider the supervised training of DNNs, which arises in many of the above applications. We focus on the central problem of optimizing the weights of the given DNN such that it accurately approximates the relation between observed input and target data. Devising effective solvers for this optimization problem is notoriously challenging due to the large number of weights, nonconvexity, data sparsity, and nontrivial choice of hyperparameters. To solve the optimization problem more efficiently, we propose the use of variable projection (VarPro), a method originally designed for separable nonlinear least-squares problems. Our main contribution is the Gauss--Newton VarPro method (GNvpro) that extends the reach of the VarPro idea to nonquadratic objective functions, most notably cross-entropy loss functions arising in classification. These extensions make GNvpro applicable to all training problems that involve a DNN whose last layer is an affine mapping, which is common in many state-of-the-art architectures. In our four numerical experiments from surrogate modeling, segmentation, and classification, GNvpro solves the optimization problem more efficiently than commonly used stochastic gradient descent (SGD) schemes. Finally, GNvpro finds solutions that generalize well, and in all but one example better than well-tuned SGD methods, to unseen data points.

97 MATHEMATICS AND COMPUTING↗