SEARCH · Search NASA
Results for “Neural Network Models”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Development of message passing-based graph convolutional networks for classifying cancer pathology reports
Abstract Background Applying graph convolutional networks (GCN) to the classification of free-form natural language texts leveraged by graph-of-words features (TextGCN) was studied and confirmed to be an effective means of describing complex natural language texts. However, the text classification models based on the TextGCN possess weaknesses in terms of memory consumption and model dissemination and distribution. In this paper, we present a fast message passing network (FastMPN), implementing a GCN with message passing architecture that provides versatility and flexibility by allowing trainable node embedding and edge weights, helping the GCN model find the better solution. We applied the FastMPN model to the task of clinical information extraction from cancer pathology reports, extracting the following six properties: main site, subsite, laterality, histology, behavior, and grade. Results We evaluated the clinical task performance of the FastMPN models in terms of micro- and macro-averaged F1 scores. A comparison was performed with the multi-task convolutional neural network (MT-CNN) model. Results show that the FastMPN model is equivalent to or better than the MT-CNN. Conclusions Our implementation revealed that our FastMPN model, which is based on the PyTorch platform, can train a large corpus (667,290 training samples) with 202,373 unique words in less than 3 minutes per epoch using one NVIDIA V100 hardware accelerator. Our experiments demonstrated that using this implementation, the clinical task performance scores of information extraction related to tumors from cancer pathology reports were highly competitive.
Machine Learning-Driven Reliability Estimation of PV Inverters Considering Alert-Ambient Variability
Weather-induced spatio-temporal degradation limits outdoor PV inverter lifetime and reliability, necessitating advanced data analysis. This study employs a top-down, data-driven approach utilizing multiple machine learning (ML) algorithms to estimate inverter reliability in a 1.4 MW PV power plant, considering factors such as irradiance, humidity, temperature, time of day, and weather conditions. An extensive alert dataset from 17 identical inverters, including alert types, propagation, and frequency, reveals significant correlations with environmental factors and inverter output power, enabling the construction of a performance reliability model. Dual-stage supervised-ML models are evaluated for accuracy, with the ‘classification-regression’ model by an artificial neural network (ANN) tested on the averaged “Alert-Ambient” dataset, which is outperformed by ‘clustering-regression’ models using random forest (RF) and K-Nearest Neighbors (KNN) on individual inverter datasets. K-means clustering applies principal component analysis to reduce dimensions, achieving improved accuracy beyond the 80% achieved by ANN on the averaged dataset. Second-stage regression estimates inverter reliability with a mean square error of 0.0195 on the averaged dataset and as low as 0.002 on individual inverter datasets using RF. Furthermore, these findings highlight the method's suitability for estimating PV inverter output reliability under ambient conditions, essential for digital twin development and related applications.
A novel multi-model neuro-fuzzy-based MPPT for three-phase grid-connected photovoltaic system
This paper presents a novel methodology for Maximum Power Point Tracking (MPPT) of a grid-connected 20 kW photovoltaic (PV) system using neuro-fuzzy network. The proposed method predicts the reference PV voltage guarantying optimal power transfer between the PV generator and the main utility grid. The neuro-fuzzy network is composed of a fuzzy rule-based classifier and three multi-layered feed forwarded Artificial Neural Networks (ANN). Inputs of the network (irradiance and temperature) are classified before they are fed into the appropriated ANN for either training or estimation process while the output is the reference voltage. The main advantage of the proposed methodology, comparing to a conventional single neural network-based approach, is the distinct generalization ability regarding to the nonlinear and dynamic behavior of a PV generator. In fact, the neuro-fuzzy network is a neural network based multi-model machine learning that defines a set of local models emulating the complex and nonlinear behavior of a PV generator under a wide range of operating conditions. Simulation results under several rapid irradiance variations proved that the proposed MPPT method fulfilled the highest efficiency comparing to a conventional single neural network and the Perturb and Observe (P and O) algorithm dispositive. (author)
Hybrid data-driven and model-informed online tool wear detection in milling machines
Precision machining tool wear is responsible for low product throughput and quality. Monitoring the tool wear online is vital to prevent degradation in machining quality. However, direct real-time tool wear measurement is not practical. This paper presents residual-based anomaly detection models, combining a hybrid model comprised of a physics-based model and a data-driven model (a decision tree or a neural network) to predict signals of interest (e.g., power or forces) under nominal conditions, followed by Page’s cumulative sum test for detecting tool wear on-line using the computer numerical control machine measurements. The most informative features are ranked using dynamic programming and its approximation variants from real-time measurements and machine settings, such as the width of cut, depth of cut, feed rate and spindle speed, that serve as inputs to the predictive models. The baseline nominal model is incrementally updated with experimental data via a gradient boosted adaptation model to generate the residuals that account for discrepancies between the actual machine data under normal conditions and the baseline nominal model predictions. The hybrid model is validated against 20 Mazak milling machine experimental tests and one Haas run-to-failure experiment. The proposed anomaly detector is applied to synthetic data from simulations of the physics-based model at different operating conditions, measurement noise levels, and tool wear levels, and the methods were able to achieve an overall 92% accuracy in data with 1% noise. The anomaly detection methods based on hybrid model reduced the false alarms of either the data-driven or physical-based models alone, and are found to be capable of good online detection of tool wear.
Multi-fidelity modeling to predict the rheological properties of a suspension of fibers using neural networks and Gaussian processes
Unveiling the rheological properties of fiber suspensions is of paramount interest to many industrial applications. There are multiple factors, such as fiber aspect ratio and volume fraction, that play a significant role in altering the rheological behavior of suspensions. Three-dimensional (3D) numerical simulations of coupled differential equations of the suspension of fibers are computationally expensive and time-consuming. Machine learning algorithms can be trained on the available data and make predictions for the cases where no numerical data are available. However, some widely used machine learning surrogates, such as neural networks, require a relatively large training dataset to produce accurate predictions. Multi-fidelity models, which combine high-fidelity data from numerical simulations and less expensive lower fidelity data from resources such as simplified constitutive equations, can pave the way for more accurate predictions. Here, we focus on neural networks and the Gaussian processes with two levels of fidelity, i.e., high and low fidelity networks, to predict the steady-state rheological properties, and compare them to the single-fidelity network. High-fidelity data are obtained from direct numerical simulations based on an immersed boundary method to couple the fluid and solid motion. The low-fidelity data are produced by using constitutive equations. Multiple neural networks and the Gaussian process structures are used for the hyperparameter tuning purpose. Results indicate that with the best choice of hyperparameters, both the multi-fidelity Gaussian processes and neural networks are capable of making predictions with a high level of accuracy with neural networks demonstrating marginally better performance.
Improving the Transportability of a Deep Learning Denoising Model Using Transfer Learning Techniques
The adoption of machine learning techniques in the seismology community has led to great performance improvements in several areas, including signal processing. Specifically, the development of deep learning–based seismic waveform denoising models has the potential to yield improvements in signal detection capabilities for networks operating in particularly noisy environments. Recent advancements in the design of these deep learning denoising models have included the incorporation of continuous and discrete wavelet transform functions into the network architecture to improve the learning capabilities and efficiency of said models. These wavelet transform–based seismic denoising models have shown improved denoising capabilities in regions where there is good agreement between the data features present in the training and evaluation datasets. However, questions remain about the overall transportability of these models to other monitoring regions. Here, in this study, we will determine the baseline transportability of a newly developed multilevel wavelet‐transform convolutional neural network (MWCNN) seismic denoising model. We accomplish this by taking a version of the MWCNN denoising model trained on data collected from the Utah region and evaluating its denoising performance on datasets collected from the neighboring Nevada region, which differ with regard to monitoring sensor types and event histories. We find that there is a notable variability in denoising performance related to the degree of similarity between the initial and new target datasets. The most notable difference in denoising performance is the ability of the denoising model to preserve accurate amplitude information associated with the signal energy present in the waveform data. Finally, we evaluate the ability of transfer learning techniques to improve the transportability of the MWCNN denoising model. We find that although there is still a performance gap present in the denoising results of the MWCNN model, transfer learning did yield improved results.
PINNSTRIPES (Physics-Informed Neural Network SurrogaTe for Rapidly Identifying Parameters in Energy Systems) [SWR-22-12]
Energy systems models typically take the form of complex partial differential equations which make multiple forward calculations prohibitively expensive. Fast and data-efficient construction of surrogate models is of utmost importance for applications that require parameter exploration such as design optimization and Bayesian calibration. In presence of a large number of parameters, surrogate models that capture correct dependencies may be difficult to construct with traditional techniques. The issue is addressed here with the formulation of the surrogate model constructed via Physics-Informed Neural Networks (PINN) which capture the dependence with respect to the parameters to estimate, while using a limited amount of data. Since forward evaluations of the surrogate model are cheap, parameter exploration is made inexpensive, even when considering a large number of parameters.
Linear eddy mixing based tabulation and artificial neural networks for large eddy simulations of turbulent flames
A large eddy simulation (LES) sub-grid model is developed based on the artificial neural network (ANN) approach to calculate the species instantaneous reaction rates for multi-step, multi-species chemical kinetics mechanisms. The proposed methodology depends on training the ANNs off-line on a thermo-chemical database representative of the actual composition and turbulence (but not the actual geometrical problem) of interest, and later using them to replace the stiff ODE solver (direct integration (DI)) to calculate the reaction rates in the sub-grid. The thermo-chemical database is tabulated with respect to the thermodynamic state vector without any reduction in the number of state variables. The thermo-chemistry is evolved by stand-alone linear eddy mixing (LEM) model simulations under both premixed and non-premixed conditions, where the unsteady interaction of turbulence with chemical kinetics is included as a part of the training database. The proposed methodology is tested in LES and in stand-alone LEM studies of three distinct test cases with different reduced mechanisms and conditions. LES of premixed flame-turbulence-vortex interaction provides direct comparison of the proposed ANN method against DI and ANNs trained on thermo-chemical database created using another type of tabulation method. It is shown that the ANN trained on the LEM database can capture the correct flame physics with accuracy comparable to DI, which cannot be achieved by ANN trained on a laminar premix flame database. A priori evaluation of the ANN generality within and outside its training domain is carried out using stand-alone LEM simulations as well. Results in general are satisfactory, and it is shown that the ANN provides considerable amount of memory saving and speed-up with reasonable and reliable accuracy. The speed-up is strongly affected by the stiffness of the reduced mechanism used for the computations, whereas the memory saving is considerable regardless. (author)
Autoencoder Neural Network for chemically reacting systems
Incorporating detailed chemical kinetic models is critical for accurate simulations of reacting flows. However, detailed models involve a large number of thermochemical (TC) state variables. Solving the governing equations to evolve these TC variables becomes impractical for real-world applications. In this work, we propose an autoencoder (AE) neural network (NN)-based reduced model to accelerate such simulations. The AE NN is first trained to find a low-dimensional latent representation of the TC states. Then, the evolving state of a chemical system can be tracked by solving the equations of the latent variables instead of the original TC equations. We demonstrate the reduced model in a syngas CO/H 2 combustion system, using training data collected from canonical perfectly stirred reactors (PSRs). It is found that the AE model can reduce the dimension of the combustion system from 12 to 2 while maintaining low reconstruction error and excellent elemental mass conservation for the test dataset. In the a posteriori test, the combustion states obtained from solving the two latent equations are compared to those from solving the 12 equations of the full model. The AE reduced method is found to be able to capture the diverse combustion states on the top two branches of the S-curve well including the extinction turning point, but with higher prediction errors for states near the ignition turning point.
Data optimization for large batch distributed training of deep neural networks
Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.
A hybrid CNN-LSTM surrogate model for hyper-resolution spatiotemporal flood forecasting in Norfolk, Virginia
Study region: Norfolk, Virginia, United States Study focus: Accurate and timely flood forecasting is essential for enhancing resilience in coastal urban areas in the context of increasing frequency and intensity of rainfall, sea level rise and urbanization. This study presents a hybrid deep learning-based surrogate model that integrates Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to enable real-time spatiotemporal flood forecasting. The model leverages CNN to capture spatial features from inputs such as elevation and Topographic Wetness Index (TWI), while LSTM processes time-series inputs of rainfall and tide data to capture temporal features. New hydrologic insights for the region: The hybrid CNN-LSTM model was trained using the physics-based hydrodynamic model simulations obtained from the Two-dimensional Unsteady FLOW (TUFLOW) model for Norfolk, Virginia, and achieved high predictive accuracy across diverse flood-prone areas. The reduced computational time from four to six hours using TUFLOW to 3.2 min per event using CNN-LSTM enables rapid flood inundation mapping and early warning applications. The model effectively captured both spatial flood extents and their temporal evolution across different flooding scenarios, providing forecasts at a 2.5-m spatial resolution and 15-min temporal resolution and a one-hour-ahead prediction horizon. While challenges remain in terms of transferability to new regions and real-time data assimilation, this approach demonstrates strong potential for supporting operational flood risk management in coastal urban environments.
Deep-Learning-Based Koopman Modeling for Online Control Synthesis of Nonlinear Power System Transient Dynamics
Power system stability and control have become more challenging due to the increasing uncertainty associated with renewable generation. Here, the performance of conventional control is highly driven by the physics-based offline-developed dynamic models that can deviate from the actual system characteristics under different operating conditions and/or configurations. Data-driven approaches based on online measurements can be a better solution to addressing these issues by capturing real-time operation conditions. This article describes a novel fully data-driven probabilistic framework to derive a linear representation of postcontingency grid dynamics and online prescribe control based on the derived model to enhance transient stability. The complex nonlinear power system dynamics is approximated by a linear model by using multiple neural network modules that infer distributions of the observations and introducing a Koopman layer to sample possible Koopman linear models from the inferred distributions. The trained model features linearity that can be easily incorporated into the existing linear control design paradigm and ease the controller design process. The effectiveness of Koopman-based control designs is validated through comparative case studies, which demonstrate increased prediction accuracy and control performance when applied to a power system with heterogeneous generator dynamics.
Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning
Computational modeling of chemical and biological systems at atomic resolution is a crucial tool in the chemist’s toolset. The use of computer simulations requires a balance between cost and accuracy: quantum-mechanical methods provide high accuracy but are computationally expensive and scale poorly to large systems, while classical force fields are cheap and scalable, but lack transferability to new systems. Machine learning can be used to achieve the best of both approaches. Here we train a general-purpose neural network potential (ANI-1ccx) that approaches CCSD(T)/CBS accuracy on benchmarks for reaction thermochemistry, isomerization, and drug-like molecular torsions. This is achieved by training a network to DFT data then using transfer learning techniques to retrain on a dataset of gold standard QM calculations (CCSD(T)/CBS) that optimally spans chemical space. The resulting potential is broadly applicable to materials science, biology, and chemistry, and billions of times faster than CCSD(T)/CBS calculations.
Journey Over Destination: Differentiable Sensor Placement Enhances Generalization [Poster]
The challenge of reconstructing spatial fields that change over time from limited sensor data has been a focal point for many research studies. Various machine learning methods have been used in attempts to address this complex issue, including convolutional neural networks. All of the proposed methods share a common requirement that the user needs to manually determine the sensor positions. This requirement remains a limiting factor in the ongoing quest for efficient learning and accurate field reconstruction. This study aims to present a method that enables a model to optimize sensor positions via backpropagation, thereby facilitating the model’s exploration of the spatial domain and enhancing sensor positioning effectively. Indexing naturally incorporates discrete decisions. This operation is nondifferentiable which is a requirement for the application of gradient-based optimization methods. We showcased its effectiveness by training an attention-based neural network, which achieved top-tier performance on two separate datasets. To our knowledge, this represents the first fully end-to-end differentiable workflow for enhancing sensor placement within a neural network model.
TNet: A Model-Constrained Tikhonov Network Approach for Inverse Problems
Deep learning (DL), in particular deep neural networks, by default is purely data-driven and in general does not require physics. This is the strength of DL but also one of its key limitations when applied to science and engineering problems in which underlying physical properties—such as stability, conservation, and positivity—and accuracy are required. DL methods in their original forms are often not capable of respecting the underlying mathematical models or achieving desired accuracy even in big-data regimes. On the other hand, many data-driven science and engineering problems, such as inverse problems, typically have limited experimental or observational data, and DL would overfit the data in this case. Leveraging information encoded in the underlying mathematical models, we argue, not only compensates for missing information in low data regimes but also provides opportunities to equip DL methods with the underlying physics, hence promoting better generalization. This paper develops a model-constrained DL approach and its variant TNet—a Tikhonov neural network—which are capable of learning not only information hidden in the training data but also in the underlying mathematical models to solve inverse problems governed by partial differential equations in low data regimes. We provide the constructions and some theoretical results for the proposed approaches for both linear and nonlinear inverse problems. Since TNet is designed to learn inverse solutions with Tikhonov regularization, it is interpretable: in fact it recovers Tikhonov solutions for linear cases while potentially approximating Tikhonov solutions for nonlinear inverse problems. We also prove that data randomization can enhance not only the smoothness of the networks but also their generalizations. Comprehensive numerical results confirm the theoretical findings and show that with even as little as 1 training data sample for one-dimensional (1D) deconvolution, 5 for an inverse 2D heat conductivity problem, 100 for inverse initial conditions for a time-dependent 2D Burgers’s equation, and 50 for inverse initial conditions for 2D Navier–Stokes equations, TNet solutions can be as accurate as Tikhonov solutions while being several orders of magnitude faster. Furthermore, this is possible owing to the model-constrained term, replications, and randomization.
Extrapolation of nuclear structure observables with artificial neural networks
Calculations of nuclei are often carried out in finite model spaces. Thus, finite-size corrections enter, and it is necessary to extrapolate the computed observables to infinite model spaces. In this work, we employ extrapolation methods based on artificial neural networks for observables such as the ground-state energy and the point-proton radius. We extrapolate results from no-core shell model and coupled-cluster calculations to very large model spaces and estimate uncertainties. Training the network on different data typically yields extrapolation results that cluster around distinct values. We show that a preprocessing of input data, and the inclusion of correlations among the input data, reduces the problem of multiple solutions and yields more stable extrapolated results and consistent uncertainty estimates. We perform extrapolations for ground-state energies and radii in 4 He, 6 Li, and 16 O, and compare the predictions from neural networks with results from infrared extrapolations.
Development, Demonstration and Validation of Data-Driven Compact Diode Models for Circuit Simulation and Analysis
Compact semiconductor device models are essential for efficiently designing and analyzing large circuits. However, traditional compact model development requires a large amount of manual effort and can span many years. Moreover, inclusion of new physics (e.g., radiation effects) into an existing model is not trivial and may require redevelopment from scratch. Machine Learning (ML) techniques have the potential to automate and significantly speed up the development of compact models. In addition, ML provides a range of modeling options that can be used to develop hierarchies of compact models tailored to specific circuit design stages. In this paper, we explore three such options: (1) table-based interpolation, (2) Generalized Moving Least-Squares, and (3) feedforward Deep Neural Networks, to develop compact models for a p-n junction diode. We evaluate the performance of these "data-driven" compact models by (1) comparing their voltage-current characteristics against laboratory data, and (2) building a bridge rectifier circuit using these devices, predicting the circuit's behavior using SPICE-like circuit simulations, and then comparing these predictions against laboratory measurements of the same circuit.