Search NASA⌕ Search

SEARCH · Search NASA

Results for “long short-term memory (LSTM) network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

125 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Long–short-term memory encoder–decoder with regularized hidden dynamics for fault detection in industrial processes

The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Here, data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.

42 ENGINEERING↗
DOE OSTI2023

High-Resolution Synthetic Solar Irradiance Sequence Generation: An LSTM-Based Generative Adversarial Network

The rapid growth of renewable energy resources penetration is bringing more challenges to power system planning and operation. Relevant renewable energy integration studies, such as the capability and dynamic performance of inverter-based resources' primary frequency response and fast frequency response, require high-resolution renewable generation output data that are representative of renewable energy resources. This paper focuses on creating synthetic but realistic solar irradiance data and proposes a long short-term memory-based generative adversarial network to generate high-resolution (second-level) solar irradiance sequences from low-resolution (minute-level) measurements. Combined with a classifier to recognize the solar irradiance patterns, the proposed model is trained using multi-loss functions to accurately capture the temporal correlations among both high-resolution and low-resolution sequences. Verification of the proposed approach is performed on the data set of the Oahu Solar Measurement Grid collected through the National Renewable Energy Laboratory. The results of the case studies demonstrate the proposed approach's capability to capture the statistical characteristics of different solar irradiance patterns and to generate high-quality synthetic solar irradiance sequences in high resolution.

dynamic scheduling↗
DOE OSTI2022

Differentiable, Learnable, Regionalized Process-Based Models With Multiphysical Outputs can Approach State-Of-The-Art Hydrologic Prediction Accuracy

Predictions of hydrologic variables across the entire water cycle have significant value for water resources management as well as downstream applications such as ecosystem and water quality modeling. Recently, purely data-driven deep learning models like long short-term memory (LSTM) showed seemingly insurmountable performance in modeling rainfall runoff and other geoscientific variables, yet they cannot predict untrained physical variables and remain challenging to interpret. Here, we show that differentiable, learnable, process-based models (called δ models here) can approach the performance level of LSTM for the intensively observed variable (streamflow) with regionalized parameterization. We use a simple hydrologic model HBV as the backbone and use embedded neural networks, which can only be trained in a differentiable programming framework, to parameterize, enhance, or replace the process-based model's modules. Without using an ensemble or post-processor, δ models can obtain a median Nash-Sutcliffe efficiency of 0.732 for 671 basins across the USA for the Daymet forcing data set, compared to 0.748 from a state-of-the-art LSTM model with the same setup. For another forcing data set, the difference is even smaller: 0.715 versus 0.722. Meanwhile, the resulting learnable process-based models can output a full set of untrained variables, for example, soil and groundwater storage, snowpack, evapotranspiration, and baseflow, and can later be constrained by their observations. Both simulated evapotranspiration and fraction of discharge from baseflow agreed decently with alternative estimates. The general framework can work with models with various process complexity and opens up the path for learning physics from big data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32 x 32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Securing Smart Manufacturing: Detection of Cyber-Physical Attacks in CNC-Based Systems

As Industry 4.0 advances, the integration of computer numerical control (CNC) machines and advanced manufacturing technologies is transforming production into smart manufacturing systems that blend physical and digital processes as cyber-physical systems. However, this increased cyber-physical connectivity exposes manufacturing systems to cyber threats that can cause severe operational and financial disruptions. This paper presents a comparative study on cyber attacks and anomaly detection techniques in manufacturing, focusing on network traffic from CNC machines. The data extracted from network packets includes machine commands and control signals exchanged between the machine's interface and control system, crucial for maintaining operational integrity. We explore two types of cyber attacks, design modification and command injection, which pose substantial risks to CNC machine productivity and system integrity. Our investigation involves experiments on a real CNC system, highlighting the urgent need for effective detection mechanisms. To address these threats, we evaluate three anomaly detection methods: dynamic time warping (DTW), rolling average, and a deep learning, long short-term memory (LSTM) time-series-based autoencoder. Each is assessed for its effectiveness in identifying anomalous behaviors caused by the attacks. Our findings demonstrate the unique strengths and limitations of each detection technique, providing a deeper understanding of their applicability in realworld manufacturing environments. The comparative analysis indicates that while certain methods are highly effective against specific attack types, others offer broader applicability across different attacks. This study contributes to the accurate detection of anomalies in CNC machining processes, thereby enhancing the reliability and security of smart manufacturing systems against diverse cyber threats.

Williams, Bethanie [Tennessee Technological Univer↗
DOE OSTI2024

Hydrologic connectivity and dynamics of solute transport in a mountain stream: Insights from a long-term tracer test and multiscale transport modeling informed by machine learning

The movement of solutes in a watershed is a complex process with multiple interactions and feedbacks across spatial and temporal scales. Modeling the dynamics of solute transport along diverse hydrologic pathways within watersheds – from hillslopes to stream channels and in and out of the hyporheic zones – is challenging but critically important, as these processes integrate and contribute to the biogeochemical functioning of the river corridor up to the river network scale. Here we use results from a long-term network-scale tracer test at the H.J. Andrews experimental forest in western Cascade Mountains, Oregon, USA to inform a multiscale framework for transport in stream corridors. The framework uses a Lagrangian-based subgrid model to represent the effects of hyporheic exchange flow and advective transport at stream network scales. The spatially and temporally resolved stream discharge needed for the transport model is imputed across the river system by an entity-aware long short-term memory network. Modeled concentrations show good agreements with the observations and exhibit power scaling laws indicative of a very wide range of timescales over which hyporheic exchange flow occurs. Our results demonstrate a data-informed modeling framework that links dynamical processes occurring at small scales to a network context to help understand how changes at reach scale cascade into network-scale effects, providing a useful tool for sustainable river basin management.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Hardware-Based Emulator with Deep Learning Model for Building Energy Control and Prediction Based on Occupancy Sensors’ Data

Heating, ventilation, and air conditioning (HVAC) is the largest source of residential energy consumption. Occupancy sensors’ data can be used for HVAC control since it indicates the number of people in the building. HVAC and sensors form a typical cyber-physical system (CPS). In this paper, we aim to build a hardware-based emulation platform to study the occupancy data’s features, which can be further extracted by using machine learning models. In particular, we propose two hardware-based emulators to investigate the use of wired/wireless communication interfaces for occupancy sensor-based building CPS control, and the use of deep learning to predict the building energy consumption with the sensor data. We hypothesize is that the building energy consumption may be predicted by using the occupancy data collected by the sensors, and question what type of prediction model should be used to accurately predict the energy load. Another hypothesis is that an in-lab hardware/software platform could be built to emulate the occupancy sensing process. The machine learning algorithms can then be used to analyze the energy load based on the sensing data. To test the emulator, the occupancy data from the sensors is used to predict energy consumption. The synchronization scheme between sensors and the HVAC server will be discussed. We have built two hardware/software emulation platforms to investigate the sensor/HVAC integration strategies, and used an enhanced deep learning model—which has sequence-to-sequence long short-term memory (Seq2Seq LSTM)—with an attention model to predict the building energy consumption with the preservation of the intrinsic patterns. Because the long-range temporal dependencies are captured, the Seq2Seq models may provide a higher accuracy by using LSTM architectures with encoder and decoder. Meanwhile, LSTMs can capture the temporal and spatial patterns of time series data. The attention model can highlight the most relevant input information in the energy prediction by allocating the attention weights. The communication overhead between the sensors and the HVAC control server can also be alleviated via the attention mechanism, which can automatically ignore the irrelevant information and amplify the relevant information during CNN training. Our experiments and performance analysis show that, compared with the traditional LSTM neural network, the performance of the proposed method has a 30% higher prediction accuracy.

Ye, Zhijing↗
DOE OSTI2023

Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

Abstract Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers—long short-term memory and gated recurrent unit—within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Comparative Analysis of ANN and LSTM Prediction Accuracy and Cooling Energy Savings through AHU-DAT Control in an Office Building

This paper proposes the optimal algorithm for controlling the HVAC system in the target building. Previous studies have analyzed pre-selected algorithms without considering the unique data characteristics of the target building, such as location, climate conditions, and HVAC system type. To address this, we compare the accuracy of cooling load prediction using ANN and LSTM algorithms, widely used in building energy research, to determine the optimal algorithm for HVAC control in the target building. We develop a simulation model calibrated with actual data to ensure data reliability and compare the energy consumption of the existing HVAC control method and the two algorithms-based methods. Results show that the ANN algorithm, with a CV(RMSE) of 12.7%, has a higher prediction accuracy than the LSTM algorithm, CV(RMSE) of 17.3%, making it a more suitable algorithm for HVAC control. Furthermore, implementing the ANN-based approach results in a 3.2% cooling energy reduction from the optimal control of Air Handling Unit (AHU) Discharge Air Temperature (DAT) compared to the fixed DAT at 12.8 °C in a representative day. This study demonstrates that ML-based HVAC system control can effectively reduce cooling energy consumption in HVAC systems, providing an effective strategy for energy conservation and improved HVAC system efficiency.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Time-series learning of latent-space dynamics for reduced-order model closure

In this work, we study the performance of long short-term memory networks (LSTMs) and neural ordinary differential equations (NODEs) in learning latent-space representations of dynamical equations for an advection-dominated problem given by the viscous Burgers equation. Our formulation is devised in a nonintrusive manner with an equation-free evolution of dynamics in a reduced space with the latter being obtained through a proper orthogonal decomposition. In addition, we leverage the sequential nature of learning for both LSTMs and NODEs to demonstrate their capability for closure in systems that are not completely resolved in the reduced space. We assess our hypothesis for two advection-dominated problems given by the viscous Burgers equation. We observe that both LSTMs and NODEs are able to reproduce the effects of the absent scales for our test cases more effectively than does intrusive dynamics evolution through a Galerkin projection. This result empirically suggests that time-series learning techniques implicitly leverage a memory kernel for coarse-grained system closure as is suggested through the Mori–Zwanzig formalism.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Dynamic data-driven multiscale modeling for predicting the degradation of a 316L stainless steel nuclear cladding material

Here, we have developed a long short-term memory stacked ensemble (LSTM-SE) surrogate modeling approach that can provide rapid predictions of microstructural evolution and the resultant mechanical properties of American Iron and Steel Institute (AISI) 316L series stainless steel (316LSS) fuel cladding under conditions of varying temperature and radiation dose rate. To acquire training data, we developed and implemented a kinetic Monte Carlo (KMC) model to simulate precipitation kinetics of M 23 C 6 , γ', and G phases within SS316L cladding. Experimentally reported precipitation kinetics of SS316L in literature were linked to the kinetic parameters of the simulated precipitation in our KMC model. The model was then used to simulate microstructure evolution under synthetically generated treatments of varying temperature and radiation dose rate, for periods of up to 3000 hours. Changes in volume fraction, number density, and particle size of precipitates were recorded, and particle area fractions were correlated using statistical methods to develop the surrogate model. Simultaneously, the mechanical properties of the simulated microstructures were evaluated using microstructure-based finite element method (FEM) analysis to determine the elastic modulus, yield stress, ultimate tensile strength, and elongation to failure of the aged microstructures. Using this approach, our surrogate model can predict precipitation behavior within 0.25% volume fraction and mechanical properties within 6% relative error from the values predicted by the KMC and FEM models using 50 training simulations as input. The trained recurrent neural network-based model can return estimations of precipitation kinetics and mechanical properties ~1000 times faster than the physics-based codes. This work demonstrates, as a proof of concept, that reactor material service lifetimes under variable service conditions can be predicted for a statistics-based model from a practicably obtainable dataset.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Dynamic Data-Driven Multiscale Modeling for Predicting Microstructure and Mechanical Properties of 316L Stainless Steel

We have developed a long short-term memory stacked ensemble (LSTM-SE) surrogate modeling approach that can provide rapid predictions of microstructural evolution and the resultant mechanical properties of American Iron and Steel Institute (AISI) 316L series stainless steel (316LSS) fuel cladding under conditions of varying temperature and radiation dose rate. To acquire training data, we developed and implemented a kinetic Monte Carlo (KMC) model to simulate precipitation kinetics of M 23 C 6 , γ', and G phases within SS316L cladding. Experimentally reported precipitation kinetics of SS316L in literature were linked to the kinetic parameters of the simulated precipitation in our KMC model. The model was then used to simulate microstructure evolution under synthetically generated treatments of varying temperature and radiation dose rate, for periods of up to 3000 hours. Changes in volume fraction, number density, and particle size of precipitates were recorded, and particle area fractions were correlated using statistical methods to develop the surrogate model. Simultaneously, the mechanical properties of the simulated microstructures were evaluated using microstructure-based finite element method (FEM) analysis to determine the elastic modulus, yield stress, ultimate tensile strength, and elongation to failure of the aged microstructures. Using this approach, our surrogate model can predict precipitation behavior within 0.25% volume fraction and mechanical properties within 6% relative error from the values predicted by the KMC and FEM models using 50 training simulations as input. The trained recurrent neural network-based model can return estimations of precipitation kinetics and mechanical properties ~1000 times faster than the physics-based codes. This work demonstrates, as a proof of concept, that reactor material service lifetimes under variable service conditions can be predicted for a statistics-based model from a practicably obtainable dataset.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Automated Framework for Groundwater Monitoring Using DWT with LSTM and Transformers

Environmental monitoring is critical for safeguarding public health and ecological well-being. Traditional data structuring and workflow monitoring methods consume significant time and effort, hindering timely insights and effective decision-making. Our study addresses this challenge by presenting an AI framework that automates data cleaning, structuring, and modeling processes, specifically targeting applications in groundwater monitoring. By leveraging automation for data processing and model training, our framework establishes a novel and efficient paradigm for environmental monitoring, with its potential application to the vast network of over a hundred Department of Energy Environmental Management (DoE-EM) cleanup sites across the country. It analyzes data streams from a network of groundwater Internet-of-Things (IoT) sensors deployed at the Savannah River Site (SRS) for prediction modeling. This allows human experts to focus on analysis and decision-making, ultimately leading to better environmental outcomes.The framework employs multivariate time-series forecasting methods to study and model the behavior of varying chemical analytes. The continuous learning process is enabled by utilizing deep learning techniques. It allows the framework to become more nuanced in its analysis over time, adapting to the specific characteristics of the environmental site and the evolving nature of contaminant behavior. Deep learning models known for sequence modeling, LSTM, and Transformers are employed for time series forecasting. Data processing and structuring are essential components significantly impacting the final model's performance. This hypothesis was proven by presenting a comparative analysis of model performance with processed and unprocessed data. The feature engineering approach utilized was the Discrete Wavelet Transform, which works well with time series data.

Discrete Wavelet Transform (DWT)↗
DOE OSTI2022

GrainNN: A neighbor-aware long short-term memory network for predicting microstructure evolution during polycrystalline grain formation

High fidelity simulations of grain formation in alloys are an indispensable tool for process-to-mechanical-properties characterization. Such simulations, however, can be computationally expensive as they require fine spatial and temporal discretizations. Their cost becomes an obstacle to parametric studies and ensemble runs and ultimately makes downstream tasks like optimal control and uncertainty quantification challenging. To enable such downstream tasks, we introduce GrainNN, an efficient and accurate reduced-order model for epitaxial grain growth in additive manufacturing conditions. GrainNN is a sequence-to-sequence long-short-term-memory (LSTM) deep neural network that evolves the dynamics of manually crafted features. Its innovations are (1) an attention mechanism with grain-microstructure-specific transformer architecture; and (2) an overlapping combination of several clones of the network to generalize to grain configurations that are different from those used for training. This design enables GrainNN to predict grain formation for unseen physical parameters, grain number, domain size and geometry. Furthermore, GrainNN not only reconstructs the quantities of interest but also can be pointwise accurate. In our numerical experiments, we use a polycrystalline phase field method to both generate the training data and assess GrainNN. For multiparametric, ensemble simulations with many grains, GrainNN can be orders of magnitude faster than phase field simulations, while delivering 5%–15% pointwise error. Additionally, this speedup includes the cost of the phase field simulations for generating training data.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Data-driven modeling of power generation for a coal power plant under cycling

Increased penetration of renewables for power generation has negatively impacted the dynamics of conventional fossil fuel-based power plants. The power plants operating on the base load are forced to cycle, to adjust to the fluctuating power demands. This results in an inefficient operation of the coal power plants, which leads up to higher operating losses. To overcome such operational challenge associated with cycling and to develop an optimal process control, this work analyzes a set of models for predicting power generation. Moreover, the power generation is intrinsically affected by the state of the power plant components, and therefore our model development also incorporates additional power plant process variables while forecasting the power generation. We present and compare multiple state-of-the-art forecasting data-driven methods for power generation to determine the most adequate and accurate model. We also develop an interpretable attention-based transformer model to explain the importance of process variables during training and forecasting. The trained deep neural network (DNN) LSTM model has good accuracy in predicting gross power generation under various prediction horizons with/without cycling events and outperforms the other models for long-term forecasting. The DNN memory-based models show significant superiority over other state-of-the-art machine learning models for short, medium and long range predictions. The transformer-based model with attention enhances the selection of historical data for multi-horizon forecasting, and also allows to interpret the significance of internal power plant components on the power generation. This newly gained insights can be used by operation engineers to anticipate and monitor the health of power plant equipment during high cycling periods.

01 COAL, LIGNITE, AND PEAT↗
DOE OSTI2020

Air compressor load forecasting using artificial neural network

Air compressor systems are responsible for approximately 10% of the electricity consumed in United States and European Union industry. As many researches have proven the effectiveness of using Artificial Neural Network inair compressor performance prediction, there is still a need to forecast the air compressor electrical load profile. The objective of this study is to predict compressed air systems’ electrical load profile, which is valuable to industry practitioners as well as software providers in developing better practice and tools for load managementand look-ahead scheduling programs. Two artificial neural networks, Two-Layer Feed-Forward Neural Networkand Long Short-Term Memory were used to predict an air compressors electrical load. Compressors with three different control mechanisms are evaluated with a total number of 11,874 observations. Here, the forecasts were validated using out-of-sample datasets with 5-fold cross-validation. Models produced average coefficient ofdetermination values from 0.24 to 0.94, average root-mean-square errors from 0.05 kW - 5.83 kW, and meanabsolute scaled errors from 0.20 to 1.33. The results indicate that both artificial neural networks yield goodresults for compressors using variable speed drive (average R 2 = 0.8 and no naïve forecasting), only the longshort-term memory model gives acceptable results for compressors using on/off control (average R 2 = 0.82 and no naïve forecasting), and no satisfactory results are obtained for load/unload type air compressors (models constituting active forecasting).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Wattile: Probabilistic Deep Learning-based Forecasting of Building Energy Consumption [SWR-20-94]

Accurate energy forecasting is becoming critical due to many reasons: i ) optimal distributed energy resources operations and dispatch, ii) fault detection and diagnostics, and iii) meeting operational energy efficiency targets. Wattile uses deep learning (DL) for the building's short-term load forecasting application. Two specific types of neural networks called, Long Short Term Memory (LSTM) and Sequence-to-Sequence (S2S) models are used to make predictions. Forecasting models are trained using online historical weather and occupancy indicator data streams from the Intelligent Campus Program's data acquisition systems at the National Renewable Energy Laboratory (NREL) for main meters and sub-meters of multiple building types. These models use probabilistic methods to provide quantile-based forecasts in addition to nominal conditional median predictions of electricity consumption.

Frank, Stephen↗