Search NASA⌕ Search

SEARCH · Search NASA

Results for “Deep Reinforcement Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12

Integrating Overhead Camera Imagery with Reinforcement Learning to Improve Fuel Economy Through Adaptive Traffic Control

Overhead cameras have been deployed for some time as a sensor for traffic control, with many advantages over conventional inductive loop sensors in terms of robustness, ease of installation, and simplicity with respect to maintenance. We explore the utility of such imagery (in particular, fisheye lens cameras) for intelligent traffic control in real-time using reinforcement learning, with a goal of improving fuel economy for the overall intersection and grid of intersections. We summarize our initial work in data collection and approach to ascertain the ability to visually sense vehicular fuel consumption using real-world data. We then describe our deep reinforcement learning topology to test adaptive control informed by the camera capabilities. Next, we tested the proof-of-concept through a variety of simulation scenarios using the Simulation of Urban Mobility (SUMO) traffic simulator, which included different estimates on vehicle classifications, different traffic distributions, and different traffic densities. Six different control policies were tested ranging from simple timing, to heuristic visual-based methods, to actual policies learned via the reinforcement learning approach. Our results demonstrate the feasibility of the approach for controlling traffic lights for better fuel efficiency based on visual vehicle estimates from commercial, fisheye lens cameras.

Karnowski, Thomas↗

Efficient and assured reinforcement learning-based building HVAC control with heterogeneous expert-guided training

Abstract Building heating, ventilation, and air conditioning (HVAC) systems account for nearly half of building energy consumption and $$20\%$$ of total energy consumption in the US. Their operation is also crucial for ensuring the physical and mental health of building occupants. Compared with traditional model-based HVAC control methods, the recent model-free deep reinforcement learning (DRL) based methods have shown good performance while do not require the development of detailed and costly physical models. However, these model-free DRL approaches often suffer from long training time to reach a good performance, which is a major obstacle for their practical deployment. In this work, we present a systematic approach to accelerate online reinforcement learning for HVAC control by taking full advantage of the knowledge from domain experts in various forms . Specifically, the algorithm stages include learning expert functions from existing abstract physical models and from historical data via offline reinforcement learning, integrating the expert functions with rule-based guidelines, conducting training guided by the integrated expert function and performing policy initialization from distilled expert function. Moreover, to ensure that the learned DRL-based HVAC controller can effectively keep room temperature within the comfortable range for occupants, we design a runtime shielding framework to reduce the temperature violation rate and incorporate the learned controller into it. Experimental results demonstrate up to 8.8 X speedup in DRL training from our approach over previous methods, with low temperature violation rate.

Xu, Shichao↗

Deep Multi-Agent Reinforcement Learning for Real-World Signalized Traffic Corridor Control

Signalized traffic control problem has been addressed recently with deep Reinforcement Learning (RL) approaches involving diverse state, action, and reward structures. While significant progress has been noted in the literature, open challenges still remain in the areas of adaptive signal phase timing, coordination in a multi-intersection corridor setting, and consideration of real-world traffic conditions. In the context of deep RL-based problem framing, extensions are needed that enable adaptive signal phase timings in an intersection agent's action space, computationally efficient information sharing among neighboring signalized intersection agents along a corridor, and experimentation in realistic simulation environments. In this paper, we develop a deep Advantage Actor Critic (A2C) multi-agent RL (MARL) approach capturing the research extensions above and apply it within a real-world calibrated Aimsun Next traffic corridor simulation model based on traffic data from the City of Coral Gables, Florida. For a multi-intersection corridor control setting, our numerical simulation experiments with a decentralized A2C MARL algorithm applied at different time periods led to a total average corridor travel delay reduction (expressed in seconds/mile averaged over vehicles) from 4.9% to 19.9% compared to state-of-the-art actuated control.

Shuvo, Salman S. [BATTELLE (PACIFIC NW LAB)]↗

A Study on Efficient Reinforcement Learning Through Knowledge Transfer

Although Reinforcement Learning (RL) algorithms have made impressive progress in learning complex tasks over the past years, there are still prevailing short-comings and challenges. Specifically, the sample-inefficiency and limited adaptation across tasks often make classic RL techniques impractical for real-world applications despite the gained representational power when combining deep neural networks with RL, known as Deep Reinforcement Learning (DRL). Recently, a number of approaches to address those issues have emerged. Many of those solutions are based on smart DRL architectures that enhance single task algorithms with the capability to share knowledge between agents and across tasks by introducing Transfer Learning (TL) capabilities. Here this survey addresses strategies of knowledge transfer from simple parameter sharing to privacy preserving federated learning and aims at providing a general overview of the field of TL in the DRL domain, establishes a classification framework, and briefly describes representative works in the area.

97 MATHEMATICS AND COMPUTING↗

Safe Reinforcement Learning for Emergency Load Shedding of Power Systems

The paradigm shift in the electric power grid necessitates a revisit of existing control methods to ensure the grid’s security and resilience. In particular, the increased uncertainties and rapidly changing operational conditions in power systems have revealed outstanding issues in terms of either speed, adaptiveness, or scalability of the existing control methods for power systems. On the other hand, the availability of massive real-time data can provide a clearer picture of what is happening in the grid. Recently, deep reinforcement learning (RL) has been regarded and adopted as a promising approach leveraging massive data for fast and adaptive grid control. However, like most existing machine learning (ML)- based control techniques, RL control usually cannot guarantee the safety of the power systems. In this paper, we introduce a novel method for safe RL-based load shedding of power systems that can enhance the safe voltage recovery of the electric power grid after experiencing faults. Numerical simulation on the IEEE 39-bus testcase is performed to demonstrate the effectiveness of the proposed safe RL emergency control, as well as its adaptive capability to faults not seen in the training.

reinforcement learning, emergency control, power g↗

Solving high-dimensional partial differential equations using deep learning

Developing algorithms for solving high-dimensional partial differential equations (PDEs) has been an exceedingly difficult task for a long time, due to the notoriously difficult problem known as the “curse of dimensionality.” This paper introduces a deep learning-based approach that can handle general high-dimensional parabolic PDEs. To this end, the PDEs are reformulated using backward stochastic differential equations and the gradient of the unknown solution is approximated by neural networks, very much in the spirit of deep reinforcement learning with the gradient acting as the policy function. Numerical results on examples including the nonlinear Black–Scholes equation, the Hamilton–Jacobi–Bellman equation, and the Allen–Cahn equation suggest that the proposed algorithm is quite effective in high dimensions, in terms of both accuracy and cost. Furthermore, this opens up possibilities in economics, finance, operational research, and physics, by considering all participating agents, assets, resources, or particles together at the same time, instead of making ad hoc assumptions on their interrelationships.

97 MATHEMATICS AND COMPUTING↗

A resilient network recovery framework against cascading failures with deep graph learning

Because of the increasing importance and dependencies of infrastructure networks and the potential for massive cascading failures in real-world network systems, maintenance optimization to effectively reduce system performance loss caused by diverse disruptions is of significant interest among researchers and practitioners. In this work, a new recovery framework was developed to rapidly identify important system components for maintenance to improve network resilience against cascading failures. Here this work provides distinct advantages to determine an optimal maintenance priority by combining real-time network structure importance with other maintenance prioritization based on customer preference. This approach adopts structural graph embedding and deep reinforcement learning to extract real-time network topology information (such as minimum vertex cover) to update the maintenance priority during the recovery process. Based on the case studies on synthetic networks and a US airport network, the proposed recovery framework with real-time network topology awareness shows better performance than other maintenance prioritization strategies regarding resilience enhancement. This work improves the understanding of how the changing network structure influences maintenance effects. It also provides insights of the practical usefulness of advanced deep learning on helping optimal maintenance prioritization to effectively reduce the intensity and extent of cascading failures.

42 ENGINEERING↗

PWR loading pattern optimization with reinforcement learning

The core loading pattern optimization problem belongs to the class of combinatorial optimization problem and has been studied since the dawn of commercial nuclear energy industry. It is characterized by multiple objectives and constraints, with a very high number of candidate patterns, which makes it impossible to solve explicitly. Stochastic optimization methodologies including Genetic Algorithms and Simulated Annealing are used by different nuclear utilities and vendors to perform fuel cycle reload design. Nevertheless, hand-designed solutions continue to be the prevalent method in the industry. To improve the state-of-the-art core reload patterns, we aim to create a method as scalable as possible, that agrees with the designer's goal of performance and safety. To help in this task Deep Reinforcement Learning (DRL), in particular Proximal Policy Optimization is leveraged. DRL has recently experienced a strong impetus from its successes applied to games, sometimes even reaching 'super-human' performances. This paper lays out the foundation of this method and proposes to study the behavior of several hyper-parameters that influence the DRL algorithm. The algorithm is highly dependent on multiple factors such as an exploration/exploitation trade-off that manifests through different parameters such as the number of loading patterns seen and the number of samples collected before a policy update, but also the shape of the objective function derived for the core design. Experimental results also demonstrate the effectiveness of the method in finding high-quality solutions from scratch within a reasonable amount of time. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Hierarchical Flexibility Offering Strategy for Integrated Hybrid Resources in Real-time Energy Markets

This paper proposes a hierarchical model for determining the energy flexibility offering strategy of integrated hybrid resources (IHRs) in power distribution systems to participate in real-time energy markets. The proposed model utilizes the scalability, fast response time, and uncertainty observation of deep reinforcement learning (DRL) to overcome the scalability issue of operating numerous flexible resources and deliverability of energy flexibility to the real-time markets in the presence of the network constraints. To that end, the power distribution system is divided into multiple IHRs, where different types of flexible loads, energy storage systems, and solar plants with controllable inverters are operated through local IHR controllers, trained by deep deterministic policy gradient (DDPG) algorithm. Active power request and reactive power capacity of IHRs are then transmitted to a central flexibility controller, where a quadratic optimization model ensures the deliverability of the energy flexibility to the real-time energy market by satisfying the distribution network constraints. The proposed model is implemented on the 123-bus test power distribution system, demonstrating the capability of DRL-based hierarchical model for scalable operation of IHRs in order to offer deliverable energy flexibility to the real-time energy market.

Majidi, Majid↗

AbBERT: Learning Antibody Humanness via Masked Language Modeling

Understanding the degree of humanness of antibody sequences is critical to the therapeutic antibody development process to reduce the risk of failure modes like immunogenicity or poor manufacturability. We introduce AbBERT, a transformer-based language model trained on up to 20 million unpaired heavy/light chain sequences from the Observed Antibody Space database. We first validate AbBERT using a novel “multi-mask” scoring procedure to demonstrate high accuracy in predicting complementary determining regions—including the challenging hypervariable H3 region. We then demonstrate several uses of AbBERT at various points along the antibody design process. AbBERT enhances in silico antibody optimization via deep reinforcement learning by utilizing its learned embeddings as additional observations during optimization. Within a larger computational antibody design platform, AbBERT has been successfully applied as an additional design objective, where it displays strong correlations with computational tools predicting antibody structural stability. Finally, mutant antibody sequences that have been scored as unfavorable by AbBERT have shown corresponding low yields when expressed in cells. These use cases demonstrate the power of language modeling within computational antibody design.

Bioinformatics↗

Integrated path planning and control through proximal policy optimization for a marine current turbine

This paper presents an integrated path planning and tracking control framework for a marine current turbine (MCT), where the MCT is treated as an energy-harvesting autonomous underwater vehicle (AUV). Considering the ocean (space of action) is continuous, the proposed framework employs two modules to address path planning and path tracking enabled by the proximal policy optimization (PPO) algorithm, which is a policy gradient deep reinforcement learning (RL) method. Further, to enable fully autonomous operation in a stochastic oceanic environment, the proposed path planning seeks a primary objective of maximizing the harvested energy; then, the path tracking module is designed to minimize the tracking error and avoid collisions with static and dynamic obstacles. Using field-collected acoustic Doppler current profiler (ADCP) data, the performance of the proposed framework is evaluated. Comparative studies with baseline algorithms in three different scenarios of path planning, path tracking without an obstacle, and path tracking with collision avoidance verify the effectiveness of our proposed approach.

16 TIDAL AND WAVE POWER↗

Differentiable stochastic halo occupation distribution

ABSTRACT In this work, we demonstrate how differentiable stochastic sampling techniques developed in the context of deep reinforcement learning can be used to perform efficient parameter inference over stochastic, simulation-based, forward models. As a particular example, we focus on the problem of estimating parameters of halo occupation distribution (HOD) models that are used to connect galaxies with their dark matter haloes. Using a combination of continuous relaxation and gradient re-parametrization techniques, we can obtain well-defined gradients with respect to HOD parameters through discrete galaxy catalogue realizations. Having access to these gradients allows us to leverage efficient sampling schemes, such as Hamiltonian Monte Carlo, and greatly speed up parameter inference. We demonstrate our technique on a mock galaxy catalogue generated from the Bolshoi simulation using a standard HOD model and find near-identical posteriors as standard Markov chain Monte Carlo techniques with an increase of ∼8× in convergence efficiency. Our differentiable HOD model also has broad applications in full forward model approaches to cosmic structure and cosmological analysis.

79 ASTRONOMY AND ASTROPHYSICS↗

Traffic Control via Connected and Automated Vehicles (CAVs): An Open-Road Field Experiment with 100 CAVs

The CIRCLES project aims to reduce instabilities in traffic flow, which are naturally occurring phenomena due to human driving behavior. Also called “phantom jams” or “stop-and-go waves,” these instabilities are a significant source of wasted energy. Toward this goal, the CIRCLES project designed a control system, referred to as the MegaController by the CIRCLES team, that could be deployed in real traffic. Our field experiment, the MegaVanderTest (MVT), leveraged a heterogeneous fleet of 100 longitudinally controlled vehicles as Lagrangian traffic actuators, each of which ran a controller with the architecture described in this article. The MegaController is a hierarchical control architecture that consists of two main layers. The upper layer is called the Speed Planner and is a centralized optimal control algorithm. It assigns speed targets to the vehicles, conveyed through the LTE cellular network. The lower layer is a control layer, running on each vehicle. It performs local actuation by overriding the stock adaptive cruise controller, using the stock onboard sensors. The Speed Planner ingests live data feeds provided by third parties as well as data from our own control vehicles and uses both to perform the speed assignment. The architecture of the Speed Planner allows for the modular use of standard control techniques, such as optimal control, model predictive control (MPC), kernel methods, and others. The architecture of the local controller allows for the flexible implementation of local controllers. Corresponding techniques include deep reinforcement learning (RL), MPC, and explicit controllers. Depending on the vehicle architecture, all onboard sensing data can be accessed by the local controllers or only some. Likewise, control inputs vary across different automakers, with inputs ranging from torque or acceleration requests for some cars to electronic selection of adaptive cruise control (ACC) setpoints in others. The proposed architecture technically allows for the combination of all possible settings proposed previously, that is {Speed Planner algorithms} × {local Vehicle Controller algorithms} × {full or partial sensing} × {torque or speed control}. As a result, most configurations were tested throughout the ramp up to the MegaVandertest (MVT).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Post-Disaster Microgrid Formation for Enhanced Distribution System Resilience

This paper proposes a deep reinforcement learning (DRL) based approach for post-disaster critical load restoration in active distribution systems to form microgrids through network reconfiguration to minimize critical load curtailments. Distribution networks are represented as graph networks, and optimal network configurations with microgrids are obtained by searching for the optimal spanning forest. The constraints to the research question being explored are the radial topology and power balance. Unlike existing analytical and population-based approaches, which necessitate the repetition of entire analyses and computation for each outage scenario to find the optimal spanning forest, the proposed approach, once properly trained, can quickly determine the optimal, or near-optimal, spanning forest even when outage scenarios change. When multiple lines fail in the system, the proposed approach forms microgrids with distributed energy resources in active distribution systems to reduce critical load curtailment. The proposed DRL-based model learns the action-value function using the REINFORCE algorithm, which is a model-free reinforcement learning technique based on stochastic policy gradients. A case study was conducted on a 33-node distribution test system, demonstrating the effectiveness of the proposed approach for post-disaster critical load restoration.

active distribution systems↗

Automated Control of Transactive HVACs in Energy Distribution Systems

Heating, Ventilation, and Air Conditioning (HVAC) systems contribute significantly to a building’s energy consumption. In the recent years, there is an increased interest in developing transactive approaches which could enable automated and flexible scheduling of HVAC systems based on the customer demand and the electricity prices decided by the suppliers. Flexible and automated scheduling of the HVAC systems make it a prime source for participation in residential demand response or transactive energy systems. Therefore, it is of significant interest to identify an optimal strategy to control the HVAC systems. Here, reducing the energy cost while keeping the comfort level acceptable to the users, we argue that such a control strategy should consider both the energy cost and user comfort simultaneously. Accordingly, we develop the control strategy through the solution of an optimization problem that balances between the energy cost and consumer’s dissatisfaction. This optimization enables us to solve a decision-making problem through first price prediction and then choosing HVAC temperature settings throughout the day based on the predicted price, history of the price and HVAC settings, and outside temperature. More specifically, we formulate the control design as a Markov decision process (MDP) using deep neural networks and use Deep Deterministic Policy Gradients (DDPG)-based deep reinforcement learning algorithm to find the optimal control strategy for HVAC systems that balances between electricity cost and user comfort.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Deep Reinforcement Scheduling of Energy Storage Systems for Real-time Voltage Regulation in Unbalanced LV Networks with High PV Penetration

The ever-growing higher penetration of distributed energy resources (DERs) in low-voltage (LV) distribution systems brings both opportunities and challenges to voltage support and regulation. This paper proposes a deep reinforcement learning (DRL)-based scheduling scheme of energy storage systems (ESSs) to mitigate system voltage deviations in unbalanced LV distribution networks. The ESS-based voltage regulation problem is formulated as a multi-stage quadratic stochastic program, with the objective of minimizing the expected total daily voltage regulation cost while satisfying operational constraints. While existing voltage regulation methods are mostly focused on onetime- step control, this paper explores a day-horizon systemwide voltage regulation problem. In other words, the size of action and state spaces are extremely high-dimensional and need to be delicately handled. Furthermore, in order to overcome the difficulty of modeling uncertainties and develop a realtime solution, a learn-to-schedule feedback control framework is proposed by adapting the problem to a model-free DRL setting. The proposed algorithm is tested on a customized 6-bus system and a modified IEEE 34-bus system. Simulation results validate the effectiveness and near-optimality of voltage regulation by ESS in comparison with a deterministic quadratic program solution.

Wang, Shengyi↗

FlexDRL v1.0

FlexDRL is a framework that can be used as a benchmarking tool to evaluate different Deep Reinforcement Learning algorithms in a building energy management setting. OpenAI Gym was used as wrapper to the co-simulation environment (EnergyPlus for envelope and HVAC and Modelica for battery and PV) to interface with DRL algorithms, which are developed in PyTorch. In order to facilitate the usage of FlexDRL, all the required dependencies have been packaged into a Docker container.

Touzani, Samir↗

Digital Real-Time Simulation and Power Quality Analysis of a Hydrogen-Generating Nuclear-Renewable Integrated Energy System

This paper investigates the challenges and solutions associated with integrating a hydrogen-generating nuclear-renewable integrated energy system (NR-IES) under a transactive energy framework. The proposed system directs excess nuclear power to hydrogen production during periods of low grid demand while utilizing renewables to maintain grid stability. Using digital real-time simulation (DRTS) in the Typhoon HIL 404 model, the dynamic interactions between nuclear power plants, electrolyzers, and power grids are analyzed to mitigate issues such as harmonic distortion, power quality degradation, and low power factor caused by large non-linear loads. A three-phase power conversion system is modeled using the Typhoon HIL 404 model and includes a generator, a variable load, an electrolyzer, and power filters. Active harmonic filters (AHFs) and hybrid active power filters (HAPFs) are implemented to address harmonic mitigation and reactive power compensation. The results reveal that the HAPF topology effectively balances cost efficiency and performance and significantly reduces active filter current requirements compared to AHF-only systems. During maximum electrolyzer operation at 4 MW, the grid frequency dropped below 59.3 Hz without filtering; however, the implementation of power filters successfully restored the frequency to 59.9 Hz, demonstrating its effectiveness in maintaining grid stability. Future work will focus on integrating a deep reinforcement learning (DRL) framework with real-time simulation and optimizing real-time power dispatch, thus enabling a scalable, efficient NR-IES for sustainable energy markets.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗