Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

An Interpretable Planning Bot for Pancreas Stereotactic Body Radiation Therapy

Pancreas stereotactic body radiation therapy (SBRT) treatment planning requires planners to make sequential, time-consuming interactions with the treatment planning system to reach the optimal dose distribution. We sought to develop a reinforcement learning (RL)–based planning bot to systematically address complex tradeoffs and achieve high plan quality consistently and efficiently.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Hybrid Imitation Learning for Real-Time Service Restoration in Resilient Distribution Systems

Self-healing capability is a critical factor for a resilient distribution system, which requires intelligent agents to automatically perform service restoration online, including network reconfiguration and reactive power dispatch. Here, the article proposes the imitation learning framework for training such an agent, where the agent will interact with an expert built based on the mixed-integer program to learn its optimal policy, and therefore significantly improve the training efficiency compared with exploration-dominant reinforcement learning (RL) methods. This significantly improved training efficiency makes the training problem under N-k scenarios tractable. A hybrid policy network is proposed to handle tie-line operations and reactive power dispatch simultaneously to further improve the restoration performance. The 33-bus and 119-bus systems with N-k disturbances are employed to conduct the training. The results indicate that the proposed method outperforms traditional RL algorithms such as the deep-Q network.

42 ENGINEERING↗

Chapter Nine - Automated Optimal Control in Energy Systems: The Reinforcement Learning Approach

With the development of smart grid technologies an increasing number of new devices and participants have joined modern energy systems and are inevitably making them more complicated and interdependent than ever. Optimally controlling such a complex energy system and maintaining its operation in a high-efficient, secure, and resilient manner are challenging tasks to the system operators. Fortunately, the revolution in deep learning and artificial intelligence (AI), both from hardware and algorithms perspectives, has provided new ideas and solutions to many previously intractable problems. As a result, this advance in computer science also sparked great research interests in utilizing AI in solving engineering problems related to the modern energy systems. Among many AI techniques, deep reinforcement learning (DRL) has demonstrated great potential for solving sequential optimization problems, which are very common in the engineering domains. Its ability to handle nonlinearity and stochasticity in controlled systems has out-competed many traditional optimal control algorithms. Therefore in this chapter, we focus on the state-of-the-art of DRL concepts and related algorithms, compare their pros and cons with traditional optimal control approaches and discuss the typical workflow for leveraging RL in solving complex problems in modern energy systems.

artificial intelligence↗

Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

This paper delves into designing feedback control gains for a continuous-time linear quadratic regulator (LQR) problem that is constrained to certain predefined structure with unknown state matrix. We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees in order to design these structured gains using the trajectory measurements of states and controls. Here we first formulate a model-based framework using dynamic programming (DP) to embed the structural constraint to the LQR gain computation in the continuous-time setting, and then subsequently, formulate a policy iteration RL algorithm that can alleviate the requirement of known state matrix in conjunction with maintaining the feedback gain structure. The design enables a distributed learning control design which is necessary for many large-scale cyber-physical systems. Theoretical guarantees are provided for stability and convergence of the structured reinforcement learning (SRL) algorithm. We validate our theoretical results with numerical simulations on a multi-agent networked linear time-invariant (LTI) dynamic system.

42 ENGINEERING↗

Enhancing Distribution System Resilience: A First-Order Meta-RL Algorithm for Critical Load Restoration

The increasing frequency of extreme events and the integration of distributed energy resources (DERs) into modern grids have elevated the need for resilient and efficient critical load restoration strategies in distribution systems. However, the stochastic nature of renewable DERs, limited energy resource availability and the intricate nonlinearities inherent in complex grid control problem make the problem challenging. Although reinforcement learning (RL) and warm-start RL methods have shown promising results, their performance often falls short in rapidly adapting to new, unseen situations and typically requires exhaustive problem-specific tuning. To address these gaps, we propose a First-Order Meta-based RL (FOM-RL) algorithm within an online framework for adaptive and robust critical load restoration. By harnessing local DERs as the enabling technology, FOM-RL allows the RL agent to swiftly adapt to new unseen scenarios by leveraging previously acquired knowledge of different tasks. Experimental results provide evidence that proposed algorithm learns more efficiently and showcases generalization capabilities across diverse set of operational scenarios. Moreover, a rigorous theoretical analysis yields a tight sublinear regret bound, sensitive to temporal variability, with a task-averaged optimality gap bounded by O(VM+D*/(Tsquare root(M))). These results suggest that optimality improves with task similarity and an increased number of tasks M, reaffirming the efficacy and scalability of the proposed approach in addressing the complexities of critical load restoration in distribution systems.

complexity theory↗

Comprehensive assessment of deep reinforcement learning approaches for economic dispatch in nuclear-driven microgrids

As the electrical grid integrates more variable renewable energy sources such as wind and solar, the demand for distributed and flexible systems to address this increased variability becomes critical. Nuclear-driven microgrids provide a promising solution by offering stable generation to complement intermittent renewables, ensuring grid reliability and operating efficiency. This paper proposes a recurrent deep reinforcement learning framework for optimal economic dispatch in a nuclear-powered microgrid integrating renewable energy sources, small modular reactors, battery storage systems, and balance-of-plant dynamics. A three-agent control architecture is developed, where demand and renewable energy agents act as forecasters, and a reinforcement learning-based dispatch agent performs real-time energy allocation. A nonlinear programming formulation is first used to generate an optimal baseline for benchmarking. The proposed dispatch controller, based on Proximal Policy Optimization enhanced with Long Short-Term Memory networks, exploits temporal correlations in system dynamics by taking advantage of the time series used as inputs to improve policy robustness under uncertainty. Comparative analysis against established deep reinforcement learning methods, including Proximal Policy Optimization with a feedforward architecture, Soft Actor-Critic, and Twin Delayed Deep Deterministic Policy Gradient, demonstrates superior performance. Numerical results indicate that the proposed controller achieves a 0.39% cost reduction relative to the nonlinear programming benchmark and outperforms other learning-based methods by generating additional revenue of up to 0.35%. All reinforcement learning controllers compute dispatch actions in less than 0.3 s, resulting in a computational speedup of more than three orders of magnitude over the nonlinear programming baseline. The findings of this paper highlight their applicability for real-time operation and control in nuclear-integrated microgrids under volatile operating conditions.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Soft Actor Critic Based Volt-VAR Co-optimization in Active Distribution Grids

Modern distribution networks are undergoing several technical challenges, such as voltage fluctuations, because of high penetration of distributed energy resources (DERs). This paper proposes a deep reinforcement learning (DRL)-based Volt VAR co-optimization technique for reducing voltage fluctuations as well as power loss under high penetration of DERs. In addition, the proposed approach minimizes the operational cost of the grid. A stochastic policy optimization based soft actor critic (SAC) agent is proposed to configure the optimal set-points of the reactive power outputs of the inverters. The performance of the proposed model is verified on the modified IEEE 34- and 123-bus systems and compared with a base case scenario with no reactive supply by inverters, and a local droop control approach. The results demonstrate that the proposed framework outperforms the conventional droop control method in improving the voltage profile, minimizing the network power loss, and reducing grid operational cost.

—Distribution grids, deep reinforcement learning, ↗

Reinforcement Learning for Load-balanced Parallel Particle Tracing

We explore an online reinforcement learning (RL) paradigm to dynamically optimize parallel particle tracing performance in distributed-memory systems. Our method combines three novel components: (1) a work donation algorithm, (2) a high-order workload estimation model, and (3) a communication cost model. First, we design an RL-based work donation algorithm. Our algorithm monitors workloads of processes and creates RL agents to donate data blocks and particles from high-workload processes to low-workload processes to minimize program execution time. The agents learn the donation strategy on the fly based on reward and cost functions designed to consider processes' workload changes and data transfer costs of donation actions. Second, we propose a workload estimation model, helping RL agents estimate the workload distribution of processes in future computations. Third, we design a communication cost model that considers both block and particle data exchange costs, helping RL agents make effective decisions with minimized communication costs. We demonstrate that our algorithm adapts to different flow behaviors in large-scale fluid dynamics, ocean, and weather simulation data. Our algorithm improves parallel particle tracing performance in terms of parallel efficiency, load balance, and costs of I/O and communication for evaluations with up to 16,384 processors.

Distributed and parallel particle tracing↗

Distributed Control with Collective Intelligence

We consider systems of interacting reinforcement learning (RL) algorithms that do not work at cross purposes , in that their collective behavior maximizes a global utility function. We call such systems COllective INtelligences (COINs). We present the theory of designing COINs. Then we present experiments validating that theory in the context of two distributed control problems: We show that COINs perform near-optimally in a difficult variant of Arthur's bar problem [Arthur] (and in particular avoid the tragedy of the commons for that problem), and we also illustrate optimal performance in the master-slave problem.

Wolpert, David H.↗

Decentralized Voltage Control of Large-Scale Distribution System with PVs Based on MADRL

This paper proposes a model-free decentralized control framework for the voltage regulation of large-scale distribution systems through the coordinated control of PV inverters. This is achieved by developing a novel interaction mechanism between the surrogate model and the centralized training and decentralized execution multiagent deep reinforcement learning framework. Specifically, the sparse Gaussian processes regression method is first utilized to develop the surrogate model of the original distribution system for reward calculation during the training stage, where each agent represents a sub-region in the centralized fashion for coordination strategy learning. After that, the learned control rules are used to inform controllers within each sub-region for real-time decisions with only local measurements. Comparative tests among various methods on the EPRI Ckt5 test system demonstrate the effectiveness of the proposed method.

distribution system↗

A Dynamic Pricing Method to Manage the Impact of EV Charging on the Grid Using RL

This work addresses the challenge of managing electrical vehicle (EV) charging loads on distribution feeders with the increase in deployment of fast charging stations. To mitigate the adverse impacts on feeder health, a novel dynamic grid-informed pricing approach is proposed. This approach leverages reinforcement learning (RL) to determine hourly charging prices based on real-time grid conditions. A synthetic environment was developed to train the reinforcement learning agent. A model of an IEEE 34-bus distribution feeder with EV charging stations has been developed in OpenDSS utilizing Caldera for realistic EV charging profiles. Test cases demonstrate that the dynamic pricing strategy achieves higher energy delivery to the EV end user at a lower cost compared to constant pricing methods, while lowering voltage deviations and congestion. This approach offers more granular price adjustments, responding dynamically to feeder conditions and potentially improving grid stability and efficiency. The communication architecture to implement this dynamic pricing method is described. This research contributes to the development of smart grid-informed charging solutions that can reduce the cost of charging to the end user while also helping the grid.

EV charging, dynamic pricing, grid-informed chargi↗

A dynamic pricing method to manage the impact of EV charging on the grid using RL

This work addresses the challenge of managing electrical vehicle (EV) charging loads on distribution feeders with the increase in deployment of fast charging stations. To mitigate the adverse impacts on feeder health, a novel dynamic grid-informed pricing approach is proposed. This approach leverages reinforcement learning (RL) to determine hourly charging prices based on real-time grid conditions. A synthetic environment was developed to train the reinforcement learning agent. A model of an IEEE 34-bus distribution feeder with EV charging stations has been developed in OpenDSS utilizing Caldera for realistic EV charging profiles. Test cases demonstrate that the dynamic pricing strategy achieves higher energy delivery to the EV end user at a lower cost compared to constant pricing methods, while lowering voltage deviations and congestion. This approach offers more granular price adjustments, responding dynamically to feeder conditions and potentially improving grid stability and efficiency. The communication architecture to implement this dynamic pricing method is described. This research contributes to the development of smart grid-informed charging solutions that can reduce the cost of charging to the end user while also helping the grid.

24 - POWER TRANSMISSION AND DISTRIBUTION↗

Data-driven Optimal Control Strategy for Virtual Synchronous Generator via Deep Reinforcement Learning Approach

This paper aims at developing a data-driven optimal control strategy for virtual synchronous generator (VSG) in the scenario where no expert knowledge or requirement for system model is available. Firstly, the optimal and adaptive control problem for VSG is transformed into a reinforcement learning task. Specifically, the control variables, i.e., virtual inertia and damping factor, are defined as the actions. Meanwhile, the active power output, angular frequency and its derivative are considered as the observations. Moreover, the reward mechanism is designed based on three preset characteristic functions to quantify the control targets: (1) maintaining the deviation of angular frequency within special limits; (2) preserving well-damped oscillations for both the angular frequency and active power output; (3) obtaining slow frequency drop in the transient process. Next, to maximize the cumulative rewards, a decentralized deep policy gradient algorithm, which features model-free and faster convergence, is developed and employed to find the optimal control policy. With this effort, a data-driven adaptive VSG controller can be obtained. By using the proposed controller, the inverter-based distributed generator can adaptively adjust its control variables based on current observations to fulfill the expected targets in model-free fashion. Finally, simulation results validate the feasibility and effectiveness of the proposed approach.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Towards intelligent emergency control for large-scale power systems: Convergence of learning, physics, computing and control

Here, this paper has delved into the pressing need for intelligent emergency control in large-scale power systems, which are experiencing significant transformations and are operating closer to their limits with more uncertainties. Learning-based control methods are promising and have shown effectiveness for intelligent power system control. However, when they are applied to large-scale power systems, there are multifaceted challenges such as scalability, adaptiveness, and security posed by the complex power system landscape, which demand comprehensive solutions. The paper first proposes and instantiates a convergence framework for integrating power systems physics, machine learning, advanced computing, and grid control to realize intelligent grid control at a large scale. Our developed methods and platform based on the convergence framework have been applied to a large (more than 3000 buses) Texas power system, and tested with 56 000 scenarios. Our work achieved a 26% reduction in load shedding on average and outperformed existing rule-based control in 99.7% of the test scenarios. The results demonstrated the potential of the proposed convergence framework and DRL-based intelligent control for the future grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A Q-Learning-Based Approach for Distributed Beam Scheduling in mmWave Networks

We consider the problem of distributed downlink beam scheduling and power allocation for millimeter-Wave (mmWave) cellular networks where multiple base stations (BSs) belonging to different service operators share the same unlicensed spectrum with no centralized control or explicit coordination among them. Our goal is to design efficient distributed beam scheduling and power allocation schemes such that the network-level payoff, defined as the weighted sum of the total throughput and a power penalization term, can be maximized. To this end, we propose a distributed scheduling approach to power allocation and adaptation for efficient interference management over the shared spectrum by modeling each BS as an independent Q-learning agent. As a baseline, we compare the proposed approach to the state-of-the-art non-cooperative game-based approach which was previously developed for the same problem. We conduct extensive experiments under various scenarios to verify the effect of multiple factors on the performance of both approaches. Experiment results show that the proposed approach adapts well to different interference conditions by learning from experience and thus can achieve higher payoff than the game-based approach. The proposed approach can also be integrated into our previously developed Lyapunov stochastic optimization framework for the purpose of network utility maximization with optimality guarantee. As a result, the weights in the payoff function can be automatically and optimally determined by the virtual queue values from the sub-problems derived from the Lyapunov optimization framework.

99 GENERAL AND MISCELLANEOUS↗

Convex Q-Learning in Continuous Time with Application to Dispatch of Distributed Energy Resources

Convex Q-learning is a recent approach to reinforcement learning, motivated by the possibility of a firmer theory for convergence, and the possibility of making use of greater a priori knowledge regarding policy or value function structure. This paper explores algorithm design in the continuous time domain, with a finite-horizon optimal control objective. The main contributions are (i) The new Q-ODE: a model-free characterization of the Hamilton-Jacobi-Bellman equation. (ii) A formulation of Convex Q-learning that avoids approximations appearing in prior work. The Bellman error used in the algorithm is defined by filtered measurements, which is necessary in the presence of measurement noise. (iii) Convex Q-learning with linear function approximation is a convex program. It is shown that the constraint region is bounded, subject to an exploration condition on the training input. (iv) The theory is illustrated in application to resource allocation for distributed energy resources, for which the theory is ideally suited.

Lu, Fan↗

Identification of Worst Impact Zones for Power Grids During Extreme Weather Events Using Q-Learning: Preprint

Both the frequency and intensity of extreme weather events have been trending higher in recent years, leading to significant infrastructure loss in the electric grid. The impact of these extreme weather events is desired to be analyzed and quantified in order to help transmission and distribution system operators to prepare and prevent significant losses. In this paper, we developed an approach that models the impact of extreme weather on the power grid and identifies the worst impact zone using Q-learning (a reinforcement learning approach). The identification results reveal grid vulnerability to weather events and provide insights for system operators to help achieve optimal resource allocation and crew dispatch in order to minimize the adverse impact of extreme weather. Simulation studies are conducted on the IEEE 123-node system to demonstrate the performance of the proposed approach.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Multi-agent voltage control in distribution systems using GAN-DRL-based approach

Active distribution grids can experience voltage fluctuations and violations due to the high penetration of variable distributed energy resources (DERs). These problems might occur because of the uncertain and variable generation natures of these resources, especially solar photovoltaic resources, during panel shadowing scenarios. Volt-VAR control (VVC) is an efficient method that controls the reactive power set-points of the inverters to regulate the voltage of distribution grids. Although several VVC approaches have been proposed recently, the performance of these approaches degrades significantly if behind-the-meter solar generation data are unobservable/missing. Therefore, it is necessary to impute missing/unobservable PV data accurately to be utilized in VVC approaches. Further, this paper proposes a model-free, data-driven, centrally trained, and decentrally executed multi-agent deep reinforcement learning-based VVC architecture to regulate the voltage of distribution networks. A generative adversarial network (GAN) is incorporated to impute the unobservable PV data accurately, which improves the performance of the proposed control architecture. The proposed multi-agent-soft-actor–critic algorithm (MASAC)-based VVC technique utilizes the actual PV dataset as well as the imputed dataset from the GAN framework to learn the optimal coordinated control policy for controlling the optimal reactive power set-points of PV inverters. The effectiveness of the proposed approach is analyzed on a modified IEEE 34-bus test case with added PV inverters. The results are compared and analyzed with a base case model with no VVC and VVC with a local droop control approach, genetic algorithm optimization, and a centralized soft actor–critic-based approach. Moreover, the performance of the proposed approach is compared with that of a multi-agent VVC framework without using the PV generation data and load information as the system state. The results illustrate that the proposed method with more state input improves the voltage profile and reduces the power loss of the network across various loading and PV generation scenarios.

14 SOLAR ENERGY↗