Search NASA⌕ Search

SEARCH · Search NASA

Results for “Recurrent Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

STFM: Accurate Spatio-Temporal Fusion Model for Weather Forecasting

Meteorological prediction is crucial for various sectors, including agriculture, navigation, daily life, disaster prevention, and scientific research. However, traditional numerical weather prediction (NWP) models are constrained by their high computational resource requirements, while the accuracy of deep learning models remains suboptimal. In response to these challenges, we propose a novel deep learning-based model, the Spatiotemporal Fusion Model (STFM), designed to enhance the accuracy of meteorological predictions. Our model leverages Fifth-Generation ECMWF Reanalysis (ERA5) data and introduces two key components: a spatiotemporal encoder module and a spatiotemporal fusion module. The spatiotemporal encoder integrates the strengths of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), effectively capturing both spatial and temporal dependencies. Meanwhile, the spatiotemporal fusion module employs a dual attention mechanism, decomposing spatial attention into global static attention and channel dynamic attention. This approach ensures comprehensive extraction of spatial features from meteorological data. The combination of these modules significantly improves prediction performance. Experimental results demonstrate that STFM excels in extracting spatiotemporal features from reanalysis data, yielding predictions that closely align with observed values. In comparative studies, STFM outperformed other models, achieving a 7% improvement in ground and high-altitude temperature predictions, a 5% enhancement in the prediction of the u/v components of 10 m wind speed, and an increase in the accuracy of potential height and relative humidity predictions by 3% and 1%, respectively. This enhanced performance highlights STFM’s potential to advance the accuracy and reliability of meteorological forecasting.

54 ENVIRONMENTAL SCIENCES↗

Predicting lake surface water phosphorus dynamics using process-guided machine learning

Phosphorus (P) loading to lakes is degrading the quality and usability of water globally. Accurate predictions of lake P dynamics are needed to understand whole-ecosystem P budgets, as well as the consequences of changing lake P concentrations for water quality. However, complex biophysical processes within lakes, along with limited observational data, challenge our capacity to reproduce short-term lake dynamics needed for water quality predictions, as well as long-term dynamics needed to understand broad scale controls over lake P. Here we use an emerging paradigm in modeling, process-guided machine learning (PGML), to produce a phosphorus budget for Lake Mendota (Wisconsin, USA) and to accurately predict epilimnetic phosphorus over a time range of days to decades. In our implementation of PGML, which we term a Process-Guided Recurrent Neural Network (PGRNN), we combine a process-based model for lake P with a recurrent neural network, and then constrain the predictions with ecological principles. We test independently the process-based model, the recurrent neural network, and the PGRNN to evaluate the overall approach. The process-based model accounted for most of the observed pattern in lake P; however it missed the long-term trend in lake P and had the worst performance in predicting winter and summer P in surface waters. The root mean square error (RMSE) for the process-based model, the recurrent neural network, and the PGRNN was 33.0 μg P L -1 , 22.7 μg P L -1 , and 20.7 μg P L -1 , respectively. All models performed better during summer, with RMSE values for the three models (same order) equal to 14.3 μg P L -1 , 10.9 μg P L -1 , and 10.7 μg P L -1 . Although the PGRNN had only marginally better RMSE during summer, it had lower bias and reproduced long-term decreases in lake P missed by the other two models. For all seasons and all years, the recurrent neural network had better predictions than process alone, with root mean square error (RMSE) of 23.8 μg P L -1 and 28.0 μg P L -1 , respectively. The output of PGRNN indicated that new processes related to water temperature, thermal stratification, and long term changes in external loads are needed to improve the process model. By using ecological knowledge, as well as the information content of complex data, PGML shows promise as a technique for accurate prediction in messy, real-world ecological dynamics, while providing valuable information that can improve our understanding of process.

54 ENVIRONMENTAL SCIENCES↗

Spatial-Temporal Recurrent Graph Neural Networks for Fault Diagnostics in Power Distribution Systems

Fault diagnostics are extremely important to decide proper actions toward fault isolation and system restoration. The growing integration of inverter-based distributed energy resources imposes strong influences on fault detection using traditional overcurrent relays. This paper utilizes emerging graph learning techniques to build new temporal recurrent graph neural network models for fault diagnostics. The temporal recurrent graph neural network structures can extract the spatial-temporal features from data of voltage measurement units installed at the critical buses. From these features, fault event detection, fault type/phase classification, and fault location are performed. Compared with previous works, the proposed temporal recurrent graph neural networks provide a better generalization for fault diagnostics. Moreover, the proposed scheme retrieves the voltage signals instead of current signals so that there is no need to install relays at all lines of the distribution system. Therefore, the proposed scheme is generalizable and not limited by the number of relays installed. The effectiveness of the proposed method is comprehensively evaluated on the Potsdam microgrid and IEEE 123-node system in comparison with other neural network structures.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Complexity-calibrated benchmarks for machine learning reveal when prediction algorithms succeed and mislead

Abstract Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano’s inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least $$\sim 60\%$$ ∼ 60 % higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines—specifically, large $$\epsilon$$ ϵ -machines—are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.

97 MATHEMATICS AND COMPUTING↗

Uncertainty quantification of graph convolution neural network models of evolving processes

The application of neural network models to scientific machine learning tasks has proliferated in recent years. In particular, neural networks have proved to be adept at modeling processes with spatial–temporal complexity. Nevertheless, these highly parameterized models have garnered skepticism in their ability to produce outputs with quantified error bounds over the regimes of interest. Hence there is a need to find uncertainty quantification methods that are suitable for neural networks. In this work we present comparisons of the parametric uncertainty quantification of neural networks modeling complex spatial–temporal processes with Hamiltonian Monte Carlo and Stein variational gradient descent and its projected variant. Specifically we apply these methods to graph convolutional neural network models of evolving systems modeled with recurrent neural network and neural ordinary differential equations architectures. We show that Stein variational inference is a viable alternative to Monte Carlo methods with some clear advantages for complex neural network models. For our exemplars, Stein variational interference gave similar pushed forward uncertainty profiles through time compared to Hamiltonian Monte Carlo, albeit with generally more generous variance. As a result, projected Stein variational gradient descent also produced similar uncertainty profiles to the non-projected counterpart, but large reductions in the active weight space were confounded by the stability of the neural network predictions and the convoluted likelihood landscape.

36 MATERIALS SCIENCE↗

Improved Subseasonal Forecasting of Extreme Polar Vortices Using Machine Learning

Our research was focused on forecasting the position and shape of the winter stratospheric polar vortex at a subseasonal timescale of 15 days in advance. To achieve this, we employed both statistical and neural network machine learning techniques. The analysis was performed on 42 winter seasons of reanalysis data provided by NASA giving us a total of 6,342 days of data. The state of the polar vortex for determined by using geometric moments to calculate the centroid latitude and the aspect ratio of an ellipse fit onto the vortex. Timeseries for thirty additional precursors were calculated to help improve the predictive capabilities of the algorithm. Feature importance of these precursors was performed using random forest to measure the predictive importance and the ideal number of precursors. Then, using the precursors identified as important, various statistical methods were tested for predictive accuracy with random forest and nearest neighbor performing the best. An echo state network, a type of recurrent neural network that features sparsely connected hidden layer and a reduced number of trainable parameters that allows for rapid training and testing, was also implemented for the forecasting problem. Hyperparameter tuning was performed for each methods using a subset of the training data. The algorithms were trained and tuned on the first 41 years of data, then tested for accuracy on the final year. In general, the centroid latitude of the polar vortex proved easier to predict than the aspect ratio across all algorithms. Random forest outperformed other statistical forecasting algorithms overall but struggled to predict extreme values. Forecasting from echo state network suggested a strong predictive capability past 15 days, but further work is required to fully realize the potential of recurrent neural network approaches.

54 ENVIRONMENTAL SCIENCES↗

Progress on Machine Learning for the SNS High Voltage Converter Modulators

The High-Voltage Converter Modulators (HVCM) used to power the klystrons in the Spallation Neutron Source (SNS) linac were selected as one area to explore machine learning due to reliability issues in the past and the availability of large sets of archived waveforms. Progress in the past two years has resulted in generating a significant amount of simulated and measured data for training neural network models such as recurrent neural networks, convolutional neural networks, and variational autoencoders. Applications in anomaly detection, fault classification, and prognostics of capacitor degradation were pursued in collaboration with the Jefferson Laboratory, and early promising results were achieved. This paper will discuss the progress to date and present results from these efforts.

Pappas, Chris↗

Echo state network for coarsening dynamics of charge density waves

An echo state network (ESN) is a type of reservoir computer that uses a recurrent neural network with a sparsely connected hidden layer. Compared with other recurrent neural networks, one great advantage of ESN is the simplicity of its training process. Yet, despite the seemingly restricted learnable parameters, ESN has been shown to successfully capture the spatial-temporal dynamics of complex patterns. Here we build an ESN to model the coarsening dynamics of charge-density waves (CDWs) in a semiclassical Holstein model, which exhibits a checkerboard electron density modulation at half-filling stabilized by a commensurate lattice distortion. The inputs to the ESN are local CDW order parameters in a finite neighborhood centered around a given site, while the output is the predicted CDW order of the center site at the next time step. Special care is taken in the design of couplings between hidden layer and input nodes to ensure lattice symmetries are properly incorporated into the ESN model. Since the model predictions depend only on CDW configurations of a finite domain, the ESN is scalable and transferrable in the sense that a model trained on dataset from a small system can be directly applied to dynamical simulations on larger lattices. Furthermore, our work opens avenues for efficient dynamical modeling of pattern formations in functional electron materials.

2-dimensional systems↗

Physics-Guided Deep Learning for Time-Series State Estimation Against False Data Injection Attacks

The modern power grid is a cyber-physical system. While the grid is becoming more intelligent with emerging sensing and communication techniques, new vulnerabilities are introduced and cyber security becomes a major concern. One type of cyber-attacks - False Data Injection Attacks (FDIAs) - exploits the limitations in traditional power system state estimation, and modifies system states without being detected. In this paper, we propose a physics-guided deep learning (PGDL) approach to defend against FDIAs. The PGDL takes real-time measurements as inputs to neural networks, outputs the estimated states, and reconstructs measurements considering power system physics. A deep recurrent neural network - Long Short-Term Memory (LSTM) - is employed to learn the temporal correlations among states. This hybrid learning model leads to a time-series state estimation method to defend against FDIAs. The simulation results using IEEE 14-bus test system demonstrate the accuracy and robustness of the proposed time-series state estimation under FDIAs.

Cyber security, state estimation, physics-guided d↗

Reduced‐Order Modeling of Energetic Materials Using Physics‐Aware Recurrent Convolutional Neural Networks in a Latent Space (LatentPARC)

Physics-aware deep learning (PADL) has gained popularity for use in spatiotemporal dynamics simulations, such as those in computational modeling of energetic materials (EM). We show that the challenge PADL methods face while learning complex field evolution problems can be simplified and accelerated by decoupling it into two tasks: learning complex geometric features in evolving fields and modeling dynamics over these features in a lower-dimensional feature space. We build upon our previous work on physics-aware recurrent convolutional neural networks (PARC). PARC embeds knowledge of underlying physics into its neural network architecture for more robust and accurate prediction of evolving physical fields. PARC was shown to effectively learn complex nonlinear features such as the formation of hotspots and coupled shock fronts in various initiation scenarios of EMs, as a function of microstructures, serving effectively as a microstructure-aware burn model. Here, we further accelerate PARC and reduce its computational cost by projecting the original dynamics onto a lower-dimensional invariant manifold, or “latent space.” The projected latent representation encodes the complex geometry of evolving fields (e.g., temperature and pressure) in a set of data-driven features. The reduced dimension of this latent space allows us to learn the dynamics during the initiation of EM with a lighter and more efficient model. We observe a significant decrease in training and inference time while maintaining results comparable to PARC at inference. This work takes steps towards enabling rapid prediction of EM thermomechanics at larger scales and characterization of EM structure–property–performance linkages at a full application scale.

Mathematics and Computing↗

The predictive skill of convolutional neural networks models for disease forecasting

In this paper we investigate the utility of one-dimensional convolutional neural network (CNN) models in epidemiological forecasting. Deep learning models, in particular variants of recurrent neural networks (RNNs) have been studied for ILI (Influenza-Like Illness) forecasting, and have achieved a higher forecasting skill compared to conventional models such as ARIMA. In this study, we adapt two neural networks that employ one-dimensional temporal convolutional layers as a primary building block—temporal convolutional networks and simple neural attentive meta-learners—for epidemiological forecasting. We then test them with influenza data from the US collected over 2010-2019. We find that epidemiological forecasting with CNNs is feasible, and their forecasting skill is comparable to, and at times, superior to, plain RNNs. Thus CNNs and RNNs bring the power of nonlinear transformations to purely data-driven epidemiological models, a capability that heretofore has been limited to more elaborate mechanistic/compartmental disease models.

59 BASIC BIOLOGICAL SCIENCES↗

Artificial neural network algorithms for pulse shape discrimination and recovery of piled-up pulses in organic scintillators

We developed two neural-network (NN)-based algorithms (fully-connected neural network (Fc-NN) and recurrent neural network (RNN)) to perform pulse shape discrimination (PSD) and identification of piled-up pulses produced by organic scintillators, upon interaction with neutrons and gamma rays. We tested the algorithms on measured and verification sets of data and compared their classification performances to standard approaches. At a high acquisition count rate (100,000 counts per second, cps), in the presence of a gamma-to-neutron ratio of approximately 400–1, the proposed NN-based algorithm achieves a fraction of misclassified neutron, gamma, and piled-up pulses of approximately 1%, 1.8%, and 0.6%, respectively. Compared to the traditional approach, it exhibits 3×, 14×, and 11× improved (lower) miscalculation rates for neutron, gamma, and piled-up pulses, respectively. Here, we also demonstrate the capability of NN-based algorithms of successfully recovering and identifying neutron and gamma ray compositions from piled-up pulses in challenging, high pulse count rate conditions.

42 ENGINEERING↗

Data coverage assessment on neural network based digital twins for autonomous control system

We report in a recently developed Nearly Autonomous Management and Control (NAMAC) system, neural networks (NNs) are used to develop digital twins for diagnosis (DT-Ds). However, NNs are not usually considered extrapolation models and may result in large errors if they are applied to unseen data outside the training data (uncovered). In this study, we propose a data coverage assessment (DCA) to determine if the NN-based DT-Ds are extrapolated based on their epistemic uncertainty. The uncertainty quantification algorithms and uncertainty thresholds are selected based on the confusion matrix of classifying evaluation data into covered or uncovered data. To demonstrate the adaptability of the proposed framework, we applied it to a basic feedforward neural network and a more advanced recurrent neural network based on a more nonlinear database. Case studies show that the proposed framework can distinguish unseen data for both basic and advanced applications with proper uncertainty quantification algorithms and thresholds.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Toward large-scale building detection with convolutional neural network using remote sensing images

Establishing up-to-date nationwide building maps is essential to understand urban dynamics, such as estimating population and urban planning and many other applications. However, an efficient and effective solution is yet to be developed. In this paper, for the first time we evaluate three state-of-the-art CNNs for detecting buildings across entire United States using aerial images. The three CNN architectures, fully convolutional neural network, conditional random field as recurrent neural network, and SegNet, support semantic pixel-wise labeling and focus on capturing textural information at multi-scale. We use 1-meter resolution NAIP images as the test data set, and compare the detection results across the three methods. In addition, we propose to combine signed distance function labels with SegNet, which is the preferred CNN architecture identified by our extensive evaluations. The results are further improved in terms of precision, recall rate and the number of building detected. On average, model inference on test images is less than one minute for an area of size $\sim56$ $km^2$. With these promising results and the time required to process images, the framework offers great potential toward country scale building mapping with remote sensing imagery.

Yang, Lexie↗

Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data With Spatial Information

Efficient processing of large-scale time series data is an intricate problem in machine learning. Conventional sensor signal processing pipelines with hand engineered feature extraction often involve huge computational cost with high dimensional data. Deep recurrent neural networks have shown promise in automated feature learning for improved time-series processing. However, generic deep recurrent models grow in scale and depth with increased complexity of the data. This is particularly challenging in presence of high dimensional data with temporal and spatial characteristics. Consequently, this work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to efficiently process complex multi-dimensional time series data with spatial information. Here, the cellular recurrent architecture in the proposed model allows for location-aware synchronous processing of time series data from spatially distributed sensor signal sources. Extensive trainable parameter sharing due to cellularity in the proposed architecture ensures efficiency in the use of recurrent processing units with high-dimensional inputs. This study also investigates the versatility of the proposed DCRNN model for classification of multi-class time series data from different application domains. Consequently, the proposed DCRNN architecture is evaluated using two time-series datasets: a multichannel scalp EEG dataset for seizure detection, and a machine fault detection dataset obtained in-house. The results suggest that the proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.

60 APPLIED LIFE SCIENCES↗

Dynamic machine learning-based optimization algorithm to improve boiler efficiency

With decreasing computational costs, improvement in algorithms, and the aggregation of large industrial and commercial datasets, machine learning is becoming a ubiquitous tool for process and business innovations. Machine learning is still lacking applications in the field of dynamic optimization for real-time control. This work presents a novel framework for performing constrained dynamic optimization using a recurrent neural network model combined with a metaheuristic optimizer. The framework is designed to augment an existing control system and is purely data-driven, like most industrial Model Predictive Control applications. Several recurrent neural network models are compared as well as several metaheuristic optimizers. Hyperparameters and optimizer parameters are tuned with parameter sweeps, and the resulting values are reported. Further, the best parameters for each optimizer and model combination are demonstrated in closed-loop control of a dynamic simulation, and several recommendations are made for generalizing this framework to other systems. Up to 0.953% improvement is realized over the non-optimized case for a simulated coal-fired boiler. While this is not a large improvement in percentage, the total economic impact is $991,000 per year, and this study builds a foundation for future machine learning with dynamic optimization.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Machine Learning-Based Model Predictive Control of Two-Time-Scale Systems

In this study, we present a general form of nonlinear two-time-scale systems, where singular perturbation analysis is used to separate the dynamics of the slow and fast subsystems. Machine learning techniques are utilized to approximate the dynamics of both subsystems. Specifically, a recurrent neural network (RNN) and a feedforward neural network (FNN) are used to predict the slow and fast state vectors, respectively. Moreover, we investigate the generalization error bounds for these machine learning models approximating the dynamics of two-time-scale systems. Next, under the assumption that the fast states are asymptotically stable, our focus shifts toward designing a Lyapunov-based model predictive control (LMPC) scheme that exclusively employs the RNN to predict the dynamics of the slow states. Additionally, we derive sufficient conditions to guarantee the closed-loop stability of the system under the sample-and-hold implementation of the controller. A nonlinear chemical process example is used to demonstrate the theory. In particular, two RNN models are constructed: one to model the full two-time-scale system and the other to predict solely the slow state vector. Both models are integrated within the LMPC scheme, and we compare their closed-loop performance while assessing the computational time required to execute the LMPC optimization problem.

97 MATHEMATICS AND COMPUTING↗

Recall Performance for Content-Addressable Memory Using Adiabatic Quantum Optimization

A content-addressable memory (CAM) stores key-value associations such that the key is recalled by providing its associated value. While CAM recall is traditionally performed using recurrent neural network models, we show how to solve this problem using adiabatic quantum optimization. Our approach maps the recurrent neural network to a commercially available quantum processing unit by taking advantage of the common underlying Ising spin model. We then assess the accuracy of the quantum processor to store key-value associations by quantifying recall performance against an ensemble of problem sets. We observe that different learning rules from the neural network community influence recall accuracy but performance appears to be limited by potential noise in the processor. The strong connection established between quantum processors and neural network problems supports the growing intersection of these two ideas.

97 MATHEMATICS AND COMPUTING↗