Search NASA⌕ Search

SEARCH · Search NASA

Results for “Deep Reinforcement Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

Comparative analysis of model-free and model-based HVAC control for residential demand response

In this paper, we present a comparative analysis of model-free reinforcement learning (RL) and model predictive control (MPC) approaches for intelligent control of heating, ventilation, and air-conditioning (HVAC). Deep-Q-network (DQN) is used as a candidate for model-free RL algorithm. The two control strategies were developed for residential demand-response (DR) HVAC system. We considered MPC as our golden standard to compare DQN's performance. The question we tried to answer through this work was, What % of MPC's performance can be achieved by model-free RL approach for intelligent HVAC control?. Based on our test result, RL achieved an average of ≈ 62% daily cost saving of MPC. Considering the pure optimization and model-based nature of MPC methods, the RL showed very promising performance. We believe that the interpretations derived from this comparative analysis provide useful insights to choose from various DR approaches and further enhance the performance of the RL-based methods for building energy managements.

Kurte, Kuldeep↗

Tribal Revegetation Project Final Project Report: 92-Acre Area, Area 5 Radioactive Waste Management Complex, Nevada National Security Site, Nevada

Nuwu (Southern Paiute), Newe (Western Shoshone), and Nuumu (Owens Valley Paiute) are linguistically related, Numic-speaking peoples who are part of the broader Uto-Aztecan language group. Numic peoples view the land as a holistic, living, sentient being with feelings and purpose. The land is personified with human characteristics and it needs to be experienced to be understood through “learning by doing.” Numic peoples do not support ground disturbing activities within their ancestral lands, including activities tied to the storage of low-level radioactive waste or classified materials on the NNSS, which they view as culturally inappropriate. These deep-rooted ancestral connections are the impetus for reinforcing Numic responsibility for healing disturbed areas by integrating respect and patience with consistent Tribal interaction. Tribal Ecological Knowledge (TEK) is the science of describing Tribal approaches for understanding natural resources. Numic TEK is embedded in traditional teachings learned incrementally over time though experience and it evolves through lessons learned and responses to environmental changes over millennia. Therefore, TEK can broaden and enhance Western scientific knowledge associated with revegetation, especially in highly disturbed areas. The project blended TEK with Western scientific ecological methods to create a vegetative cover within test plots on the 92-Acre Area located at the Radioactive Waste Management Complex (RWMC) located in Area 5 on the Nevada National Security Site (NNSS). The vegetated test plots were systematically created for the Department of Energy (DOE) in tandem with the existing Federal Facilities Agreement and Consent Order (FFACO) with the Nevada Department of Environmental Protection (NDEP). Three previous contractor-lead attempts at revegetation, one targeting full cover revegetation and two targeting test plot revegetation, did not achieve the anticipated results at this location. When presented to the 16 American Indian Tribal nations and affiliated groups with cultural and historical ties to the NNSS, the group appointed a Tribal Revegetation Committee (TRC) that included six expert Tribal knowledge holders to collaborate with an ethnoecologist/cultural anthropologist and two biologists. Project design, planning, seed and outplant selection, spiritual land preparation, and methodology were guided by the TRC and an ethnoecologist/cultural anthropologist from Portland State University (PSU) and biologists from Desert Research Institute (DRI). Using TEK, the TRC recommended a specific seed mixture that contained nine native plant species and three species of outplants. The revegetation effort included preparing and planting thirty 10 m × 10 m (32.8 ft × 32.8 ft) seeded plots, twelve of which also included outplants; and eight 10 m × 100 m (32.8 ft × 328 ft) plots that only received outplants, all atop a waste cell cover. The TRC and the project team creatively adapted TEK with Western scientific methods so that the planned revegetation efforts could occur within the safety and security parameters of the RWMC. Test plots were subjected to one of five soil treatments with varying combinations of straw or mulch applications, soil amendment, and/or outplant planting and one of two watering treatments (watered or unwatered). Planting was divided into two events: one in the fall season and another during the subsequent spring season based on TEK and a corresponding recommendation from the TRC. Monitoring and spiritual management was conducted by the TRC to evaluate plant progress on a monthly basis (in conjunction with the biologist and anthropologist) during each respective growing season for a period of three years after planting. This approach allowed Tribal members the opportunity to conduct traditional blessings and other culturally appropriate activities to restore balance to the land in accordance with Tribal protocols. Following TEK-guided methods, successful plant establishment from seed stock and outplants was observed in many plots. Overall, plots planted in the spring, as recommended by the TRC, showed higher rates of outplant survival and native seedling emergence than those planted in the fall. This finding is significant because it is contrary to the original guidance and previous approaches provided for planting in this region. The TRC believes the frequent co-occurrence of native seedlings near surviving outplants indicates an important symbiotic relationship understood by Tribal communities and overlooked by others. Watered outplants displayed much higher survivability than unwatered plants, even after watering was reduced after the plants were established. Soil amendments and mulch created higher densities of native plants from seed. Many native seedlings showed significant delays in germination, which is considered a normal adaptation to desert climates. Some native plants did not germinate until the third growing year, whereas others germinated during the first growing year, which demonstrates the complexity of the desert environment. Evidence of native insects, reptiles, mammals, and birds, as well as native plants that were not part of the planted species, were noted and considered culturally significant. Despite the presence of non-native plants, native outplants continued to thrive and the incidence of native plant germination from seed increased over time. These successful revegetation results where previous efforts were unsuccessful reinforce the importance of integrating regionally appropriate, TEK-guided methodology with Western science to achieve positive results and the necessity of integrating Tribal involvement in all stages of the revegetation effort. Expanded approaches coupled with Tribal knowledge and tools from Western science addressed a complex problem tied to revegetating atop a low-level radioactive waste cell. The level of Tribal participation serves as a progressive model for building collaborative relationships and addressing ecological challenges on the NNSS.

54 ENVIRONMENTAL SCIENCES↗

Optimizing and Extending the Functionality of EXARL for Scalable Reinforcement Learning [Slides]

The main goal of the Co-Design Summer School 2021 is to provide algorithmic improvements to EXARL framework by improving performance and by adding functionalities. This presentation includes an introduction to reinforcement learning and to EXARL. The researchers expanded the capability of EXARL by including additional agents like (Asynchronized) Advantage Actor Critic (A2C/A3C) and Twin Delayed Deep Deterministic Policy Gradient (TD3). They also explored algorithmic improvements such as v-trace and Prioritized Experience Replay. They found that A2C/A3C performed best with v-trace and outperformed Deep Q-Network (DQN) on both the CartPole game and the ExaBooster scientific environment. Additionally, they found that TD3 performed as good as the existing Deep Deterministic Policy Gradient (DDPG) agent and that adding Prioritized Experience Replay to DDPG accelerated convergence.

97 MATHEMATICS AND COMPUTING↗

Hybrid Imitation Learning for Real-Time Service Restoration in Resilient Distribution Systems

Self-healing capability is a critical factor for a resilient distribution system, which requires intelligent agents to automatically perform service restoration online, including network reconfiguration and reactive power dispatch. Here, the article proposes the imitation learning framework for training such an agent, where the agent will interact with an expert built based on the mixed-integer program to learn its optimal policy, and therefore significantly improve the training efficiency compared with exploration-dominant reinforcement learning (RL) methods. This significantly improved training efficiency makes the training problem under N-k scenarios tractable. A hybrid policy network is proposed to handle tie-line operations and reactive power dispatch simultaneously to further improve the restoration performance. The 33-bus and 119-bus systems with N-k disturbances are employed to conduct the training. The results indicate that the proposed method outperforms traditional RL algorithms such as the deep-Q network.

42 ENGINEERING↗

Robotic Planning under Uncertainty in Spatiotemporal Environments in Expeditionary Science

In the expeditionary sciences, spatiotemporally varying environments -- hydrothermal plumes, algal blooms, lava flows, or animal migrations -- are ubiquitous. Mobile robots are uniquely well-suited to study these dynamic, mesoscale natural environments. We formalize expeditionary science as a sequential decision-making problem, modeled using the language of partially-observable Markov decision processes (POMDPs). Solving the expeditionary science POMDP under real-world constraints requires efficient probabilistic modeling and decision-making in problems with complex dynamics and observational models. Previous work in informative path planning, adaptive sampling, and experimental design have shown compelling results, largely in static environments, using data-driven models and information-based rewards. However, these methodologies do not trivially extend to expeditionary science in spatiotemporal environments: they generally do not make use of scientific knowledge such as equations of state dynamics, they focus on information gathering as opposed to scientific task execution, and they make use of decision-making approaches that scale poorly to large, continuous problems with long planning horizons and real-time operational constraints. In this work, we discuss these and other challenges related to probabilistic modeling and decision-making in expeditionary science, and present some of our preliminary work that addresses these gaps. We ground our results in a real expeditionary science deployment of an autonomous underwater vehicle (AUV) in the deep ocean for hydrothermal vent discovery and characterization. Our concluding thoughts highlight remaining work to be done, and the challenges that merit consideration by the reinforcement learning and decision-making community.

Preston, Victoria↗

Reinforcement-Learning-Based Smart Water Heater Control: An Actual Deployment

Utilizing smart control algorithms for electric water heaters (EWHs) is essential for fully harnessing the demand response (DR) potential of EWHs. For this reason, the use of reinforcement learning (RL) algorithms for EWHs has received increasing attention in recent years. However, existing RL approaches are either simulation-based or use pretrained RL agents. To this end, this paper presents the real-world deployment of a set of model-free RL approaches that aim to minimize the electricity cost of a EWH under a time-of-use electricity pricing policy using standard DR commands (e.g., shed, load up). The experiment results showed that the RL agents can help save electricity cost in the range of 11% to 14% compared to the baseline operation. This study demonstrated that RL-based EWH controllers can be deployed in real world without any prior training and can still save electricity cost.

deep learning↗

Deep Symbolic Regression

Symbolic regression is the task of finding tractable mathematical expressions that best fit an input dataset. Deep symbolic regression (DSR) searches the space of tractable mathematical expressions by using a recurrent neural network that generates mathematical expressions. The network is trained using reinforcement learning.

Petersen, BrendenK↗

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor-critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection-diffusion field. As a result, we provide explanation for the policy behaviors using knowledge of the underlying physical process.

97 MATHEMATICS AND COMPUTING↗

Learning-based framework for sensor fault-tolerant building HVAC control with model-assisted learning

As people spend up to 87% of their time indoors, intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems in buildings are essential for maintaining occupant comfort and reducing energy consumption. These HVAC systems in smart buildings rely 'on real-time sensor readings, which in practice often suffer from various faults and could also be vulnerable to malicious attacks. Such faulty sensor inputs may lead to the violation of indoor environment requirements (e.g., temperature, humidity, etc.) and the increase of energy consumption. While many model-based approaches have been proposed in the literature for building HVAC control, it is costly to develop accurate physical models for ensuring their performance and even more challenging to address the impact of sensor faults. In this work, we present a novel learning-based framework for sensor fault-tolerant HVAC control, which includes three deep learning based components for 1) generating temperature proposals with the consideration of possible sensor faults, 2) selecting one of the proposals based on the assessment of their accuracy, and 3) applying reinforcement learning with the selected temperature proposal. Moreover, to address the challenge of training data insufficiency in building-related tasks, we propose a model-assisted learning method leveraging an abstract model of building physical dynamics. Through extensive experiments, we demonstrate that the proposed fault-tolerant HVAC control framework can significantly reduce building temperature violations under a variety of sensor fault patterns while maintaining energy efficiency.

Xu, Shichao↗

Automated Construction of Artificial Lattice Structures with Designer Electronic States

Manipulating matter with a scanning tunneling microscope (STM) enables the creation of atomically defined artificial structures that host designer quantum states. However, the time-consuming nature of the manipulation process, coupled with the sensitivity of the STM tip, constrains the exploration of diverse configurations and limits the size of the designed features. In this study, we present a reinforcement learning (RL)-based framework for creating artificial structures by spatially manipulating carbon monoxide (CO) molecules on a copper substrate by using the STM tip. The automated workflow combines molecule detection and manipulation, employing deep-learning-based object detection to locate CO molecules and linear assignment algorithms to allocate these molecules to designated target sites. We initially perform molecule maneuvering based on randomized parameter sampling for sample bias, tunneling current set point, and manipulation speed. This data set is then structured into an action trajectory used to train an RL agent. The model is subsequently deployed on the STM for real-time fine-tuning of the manipulation parameters during structure construction. Our approach incorporates path-planning protocols coupled with active drift compensation to enable atomically precise fabrication of structures with significantly reduced human input while realizing larger-scale artificial lattices with the desired electronic properties. Furthermore, using our approach, we demonstrate the automated construction of an extended artificial graphene lattice and confirm the existence of a characteristic Dirac point in its electronic structure. Further challenges regarding the RL-based structural assembly scalability are discussed.

Algorithms↗

Nanoindentation mapping defects filtration for heterogeneous materials using generative adversarial networks

Advanced composite materials with multiple phases and heterogeneous microstructure necessitate spatial mapping characterization of elastic modulus to develop constitutive relations and overall mechanical response. Such modulus mapping can be obtained using the nanoindentation technique, where the indenter tip raster over the selected microstructure region. Typically, a surface preparation procedure is done in the specimens to ensure proper contact between the indenter tip and sample surface. However, a near-perfect surface finish is unachievable in heterogeneous materials, primarily with ceramic reinforcements, due to the differential material removal rate during polishing. Thus, the nanoindenter records localized erroneous measurements due to differences in surface roughness and corresponding force response. This study establishes a novel deep learning-based strategy to rectify incorrect experimental spatial measurements acquire during nanoindentation modulus mapping. Here, the integrated bicubic interpolation and generative adversarial networks (GANs) model was trained using 14 ceramic and 18 metallic data sets, each comprising 65,536 measurements. The developed algorithm was validated against experimental measurements on four unknown specimens. The standard deviation in measured elastic modulus reduces by ~50% in ceramics and ~72% in metallic samples. This computational framework proposes a novel approach to reducing uncertainty in materials’ properties using state-of-the-art computer vision techniques.

36 MATERIALS SCIENCE↗

Micrometer: Micromechanics transformer for predicting full field mechanical responses of heterogeneous materials

Predicting mechanical responses of heterogeneous materials across scales remains a significant challenge. Traditional computational methods often struggle with complex and multiscale nature of these materials, limiting their effectiveness in real-world applications. Here, in this paper, we introduce Micrometer, a vision transformer based deep learning model designed to predict full field mechanical responses of heterogeneous materials, bridging the gap between computer vision and solid mechanics problems. We show that Micrometer, trained on a large-scale high-resolution dataset of 2D fiber-reinforced composites, can achieve state-of-the-art performance in predicting microscale strain fields across a wide range of material properties and loading conditions. Our model demonstrates accuracy and computational efficiency in applications such as computational homogenization and multiscale modeling, reducing computational time by up to two orders of magnitude compared to conventional numerical solvers while maintaining less than 1 % errors in predicting macroscale stress fields. Furthermore, we showcase Micrometer’s adaptability through transfer learning experiments on new materials with limited data, highlighting its potential to tackle diverse scenarios in computational solid mechanics. These results represent a significant step towards AI-driven innovation in materials science, addressing the limitations of traditional numerical methods and paving the way for more efficient simulations of heterogeneous materials across various industrial applications.

Composite materials↗

Safe Reinforcement Learning-Based Transient Stability Control for Islanded Microgrids With Topology Reconfiguration

This paper proposes a safe reinforcement learning (RL)-based transient stability emergency control (TSEC) method for islanded microgrids. RL requires extensive interaction with the environment to learn control strategies, hence, a data-driven approach is used as a substitute for time-consuming time-domain simulation calculations. Deep sigma point processes (DSPP), which is a Gaussian process model, is utilized to predict the normal distribution of transient stability of microgrids and to construct a transient stability chance constraint. Reward-constrained policy optimization (RCPO) can simultaneously achieve objective prediction, policy learning, and constraint cost coefficient update across multiple timescales. RCPO interacts with the DSPP-based microgrid environment through a multi-process parallel manner, greatly increasing the training speed. Case studies on a real islanded microgrid demonstrate that the proposed method can efficiently and quickly obtain the optimal emergency control strategy while adhering to all hard constraints.

14 SOLAR ENERGY↗

Variational quantum reinforcement learning via evolutionary optimization

Abstract Recent advances in classical reinforcement learning (RL) and quantum computation point to a promising direction for performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in modern quantum devices. Here, we present two frameworks for deep quantum RL tasks using gradient-free evolutionary optimization. First, we apply the amplitude encoding scheme to the Cart-Pole problem, where we demonstrate the quantum advantage of parameter saving using amplitude encoding. Second, we propose a hybrid framework where the quantum RL agents are equipped with a hybrid tensor network-variational quantum circuit (TN-VQC) architecture to handle inputs of dimensions exceeding the number of qubits. This allows us to perform quantum RL in the MiniGrid environment with 147-dimensional inputs. The hybrid TN-VQC architecture provides a natural way to perform efficient compression of the input dimension, enabling further quantum RL applications on noisy intermediate-scale quantum devices.

97 MATHEMATICS AND COMPUTING↗

Using Reinforcement Learning to Optimize Quantum Circuits in thePresence of Noise

Many quantum computing frameworks currently use noise aware algorithms for implementing quantum circuits which do not scale efficiently as the size of the hardware architecture increases. As we move towards devices which utilize more qubits, it becomes increasing more important to map quantum circuits in a way that uses resources efficiently as well as maximizes the reliability of the results of that circuit. However, as the hardware increases to the point where Quantum supremacy is attainable, it will infeasible for a brute-force algorithm to find the most optimal circuit layout for circuits of medium to large depth sizes. To this end, we will to rely on reinforcement learning (RL) as a method of building quantum circuits based on observations of the noise characteristics in its environment. In this work, we create a working reinforcement learning environment in which an agent is able to make action which will build the class of circuits which creates the GHZ state. In addition to this, we also get preliminary results of the performance of a Deep Q Neural Network, which initially does not perform as well as we believe it can. In the future, we want to improve the performance of the agent and potentially generalize this environment to more classes of circuits.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Efficient distributed continual learning for steering experiments in real-time

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs to achieve high accuracy, short runtime, and scalability. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input minibatches using unbiased, global sampling. We further propose a generalization of rehearsal buffers to support both classification and generative learning tasks, as well as more advanced rehearsal strategies (notably Dark Experience Replay, leveraging knowledge distillation). We illustrate this approach with a real-life HPC streaming application from the domain of ptychographic image reconstruction. Furthermore, we run extensive experiments on up to 128 GPUs of the ThetaGPU supercomputer to compare our approach with baselines representative of training-from-scratch (the upper bound in terms of accuracy) and incremental training (the lower bound). Results show that rehearsal-based continual learning achieves a top-5 validation accuracy close to the upper bound, while simultaneously exhibiting a runtime close to the lower bound.

Asynchronous data management↗

Remaining Useful Strength (RUS) Prediction of SiCf-SiCm Composite Materials Using Deep Learning and Acoustic Emission

Prognosis techniques for prediction of remaining useful life (RUL) are of crucial importance to the management of complex systems for they can lead to appropriate maintenance interventions and improvements in reliability. While various data-driven methods have been introduced to predict the remaining useful life (RUL) of machinery systems or batteries, no research has been reported on the remaining useful strength (RUS) prediction of silicon carbide fiber reinforced silicon carbide matrix (SiCf-SiCm) materials with pivotal role in its potential usage as a structural material in nuclear reactors and turbine engines. Knowledge of its degradation process is of the utmost importance to the manufacturers. For this purpose, two approaches based on the machine-learning techniques of random-forest (RF) and convolutional neural network (CNN) are proposed to predict the RUS of SiCf-SiCm using only acoustic emission (AE) signals generated during the material’s stress applying process. Experimental results show that the CNN models achieved better predictive performance than the RF models but the latter with expert-engineered features achieves better prediction for AE signals in the early stage of degradation. Additionally, our results demonstrate that both models can correctly predict the SiCf-SiCm RUS as evaluated by our robust testing method from which the best average root mean square error (RMSE) and Pearson correlation coefficient of 3.55 ksi units and 0.85 were obtained.

36 MATERIALS SCIENCE↗

A reinforcement learning approach to long-horizon operations, health, and maintenance supervisory control of advanced energy systems

In this work, we develop a Reinforcement Learning (RL) approach to the supervisory control problem for advanced energy systems, such as novel nuclear reactors and other demand-driven, mission-critical, and component-health-sensitive energy plants. The inclusive problem landscape considered captures the stochastic confluence of plant performance, component health evolution, power demand from the grid, diverse maintenance actions, and operator-defined goals and constraints, all considered over meaningfully long-enough reasoning horizons. Key aspects of the proposed approach are a receding horizon control-inspired technique dictating time- or event-triggered supervisory policy (re-)constructions, as well as additional capability-enabling contributions such as timescale compression, to handle long reasoning horizons and uncertainty in parts of the problem, and practical yet demonstrably-effective handling of hybrid action spaces with continuous and discrete decision variables. The resulting algorithm consists of a simulation-based RL agent constructing stochastic supervisory control policies over nontrivial action spaces and for long horizons, applying the learned policy to the system for a much shorter interval, and perpetually repeating, to construct the next long-horizon policy. That next policy will only be applied, again, for a short interval, yet originally far-in-time events move progressively closer, their associated uncertainty decreases, and new events and aspects enter the reasoning horizon. The proposed methodology bridges fundamental receding horizon concepts with the unequivocally stronger and more scalable reasoning of contemporary RL. Numerical examples using Soft Actor–Critic Deep RL illustrate the operation and efficacy of the proposed technique for a power plant tasked with health-aware load following missions in a dynamic electricity market landscape.

97 MATHEMATICS AND COMPUTING↗