Search NASA⌕ Search

SEARCH · Search NASA

Results for “safe reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A safe reinforcement learning algorithm for supervisory control of power plants

Traditional control theory-based methods require tailored engineering for each system and constant fine-tuning. In power plant control, one often needs to obtain a precise representation of the system dynamics and carefully design the control scheme accordingly. Model-free Reinforcement learning (RL) has emerged as a promising solution for control tasks due to its ability to learn from trial-and-error interactions with the environment. It eliminates the need for explicitly modeling the environment’s dynamics, which is potentially inaccurate. However, the direct imposition of state constraints in power plant control raises challenges for standard RL methods. To address this, we propose a chance-constrained RL algorithm based on Proximal Policy Optimization for supervisory control. Our method employs Lagrangian relaxation to convert the constrained optimization problem into an unconstrained objective, where trainable Lagrange multipliers enforce the state constraints. In conclusion, our approach achieves the smallest distance of violation and violation rate in a load-follow maneuver for an advanced Nuclear Power Plant design.

constrained optimization↗
DOE OSTI2021

Safe Reinforcement Learning for Emergency Load Shedding of Power Systems

The paradigm shift in the electric power grid necessitates a revisit of existing control methods to ensure the grid’s security and resilience. In particular, the increased uncertainties and rapidly changing operational conditions in power systems have revealed outstanding issues in terms of either speed, adaptiveness, or scalability of the existing control methods for power systems. On the other hand, the availability of massive real-time data can provide a clearer picture of what is happening in the grid. Recently, deep reinforcement learning (RL) has been regarded and adopted as a promising approach leveraging massive data for fast and adaptive grid control. However, like most existing machine learning (ML)- based control techniques, RL control usually cannot guarantee the safety of the power systems. In this paper, we introduce a novel method for safe RL-based load shedding of power systems that can enhance the safe voltage recovery of the electric power grid after experiencing faults. Numerical simulation on the IEEE 39-bus testcase is performed to demonstrate the effectiveness of the proposed safe RL emergency control, as well as its adaptive capability to faults not seen in the training.

reinforcement learning, emergency control, power g↗
DOE OSTI2024

Optimal Management of Grid-Interactive Efficient Buildings via Safe Reinforcement Learning

Reinforcement learning (RL)-based methods have achieved significant success in managing grid-interactive efficient buildings (GEBs). However, RL does not carry intrinsic guarantees of constraint satisfaction, which may lead to severe safety consequences. Besides, in GEB control applications, most existing safe RL approaches rely only on the regularisation parameters in neural networks or penalty of rewards, which often encounter challenges with parameter tuning and lead to catastrophic constraint violations. To provide enforced safety guarantees in controlling GEBs, this paper designs a physics-inspired safe RL method whose decision-making is enhanced through safe interaction with the environment. Different energy resources in GEBs are optimally managed to minimize energy costs and maximize customer comfort. The proposed approach can achieve strict constraint guarantees based on prior knowledge of a set of developed hard steady-state rules. Simulations on the optimal management of GEBs, including heating, ventilation, and air conditioning (HVAC), solar photovoltaics, and energy storage systems, demonstrate the effectiveness of the proposed approach.

Huo, Xiang↗
DOE OSTI2025

Safe Reinforcement Learning-Based Transient Stability Control for Islanded Microgrids With Topology Reconfiguration

This paper proposes a safe reinforcement learning (RL)-based transient stability emergency control (TSEC) method for islanded microgrids. RL requires extensive interaction with the environment to learn control strategies, hence, a data-driven approach is used as a substitute for time-consuming time-domain simulation calculations. Deep sigma point processes (DSPP), which is a Gaussian process model, is utilized to predict the normal distribution of transient stability of microgrids and to construct a transient stability chance constraint. Reward-constrained policy optimization (RCPO) can simultaneously achieve objective prediction, policy learning, and constraint cost coefficient update across multiple timescales. RCPO interacts with the DSPP-based microgrid environment through a multi-process parallel manner, greatly increasing the training speed. Case studies on a real islanded microgrid demonstrate that the proposed method can efficiently and quickly obtain the optimal emergency control strategy while adhering to all hard constraints.

14 SOLAR ENERGY↗
DOE OSTI2026

Safe Deep Reinforcement Learning for Robust Frequency and Voltage-Constrained Networked Microgrid Restoration

Here, this paper proposes a safe soft actor-critic reinforcement learning (RL) algorithm–based controller for networked microgrid restoration. It formulates the post black-start start as a finite-horizon constrained Markov decision process. The RL agent co-optimizes real and reactive power set-points for both grid-forming and grid-following inverters under explicit voltage and frequency constraints, while enforcing proper power sharing via the Mean Active Power Sharing Index (MPSI) and Mean Reactive Power Sharing Index (MQSI). Numerical results obtained on the IEEE 123-bus distribution system show that the proposed method achieves a mean voltage build-up time of 0.01 s without breaching the 5% sharing-violation budget under various load scenarios, considering MPSI and MQSI indices. These findings demonstrate that the proposed method yields fast and safe black-start schedules without resorting to heuristic penalties.

Selim, Alaa [Dartmouth College, Hanover, NH (Unite↗
DOE OSTI2026

Safe Deep Reinforcement Learning for Active Distribution System Model Predictive Control with EVs and DERs

The temporal and spatial mismatch between PV generation and electric vehicle (EV) charging and discharging may cause voltage violations in active distribution networks. Despite the widespread use of deep reinforcement learning (DRL) in power system optimization and control, it lacks guarantees on constraint satisfaction during both training and deployment. This paper proposes a Lagrangian-based safe DRL approach for model predictive control (MPC) of active distribution systems with large-scale integration of PVs, EVs, and energy storage systems (ESSs). A Transformer-LSTM time-series model is proposed to forecast EV charging demand, which is then formulated as a constraint to ensure charging requirements are met. Using this prediction, a Lagrangian-based safe soft actor-critic (SAC) framework is developed for real-time control in a three-phase unbalanced distribution system, enforcing voltage safety constraints while optimizing the cumulative net reward. By integrating the forecasting model with multi-period constraints, the proposed framework jointly coordinates PV systems, EV charging and discharging, and ESS scheduling within the MPC horizon. Numerical experiments on a modified IEEE 123-bus system with real-world data show that, under a high PV penetration scenario, the proposed method increases the net reward by 30.74% and reduces average voltage violations from 0.0011 p.u. to 0.0002 p.u. compared with standard SAC. Compared with the optimal power flow (OPF) approach, it achieves similar voltage security while yielding lower line losses. It also maintains real-time control capability, reducing operation latency to 53.21 ms per 15-minute control interval. The proposed method remains effective under varying PV/EV penetrations and load conditions.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Safe Exploration Reinforcement Learning for Load Restoration using Invalid Action Masking

This paper addresses the load restoration problem after a power outage event. Our primary proposed methodology uses a multi-agent reinforcement learning method to make the optimal sequential decisions on picking up critical loads. Typically, a negative reward is provided to discourage the agents from selecting decisions that violate physical constraints during the restoration process. However, the main disadvantage of this approach is its difficulty in applying it to large-scale systems due to the curse of dimensionality. This paper introduces the invalid action masking technique to overcome this limitation. The features of this technique include zero physical constraint violations, reduced training time, and stabilization of the explo- ration process. Simulation results are performed in IEEE 13-node and IEEE 123-node systems showing the better performance of the proposed algorithm in comparison to the conventional approaches both in terms of restored power and learning curve.

reinforcement learning, blackstart, artificial int↗
DOE OSTI2023

A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability

Increasing integration of renewable resources brings more flexibility and poses new challenges to modern power systems, leading to highly nonlinear and complex dynamics. Here, this paper aims to provide a general solution framework to traditional control problems, such as frequency control and voltage control, which attempt to maintain the stability of either synchronous generators-governed or inverter-governed systems when subjected to a disturbance and simultaneously guarantee operational constraints, providing a complete complement to existing works on control design. Building on reinforcement learning (RL) and control barrier functions, the framework includes two subsystems, i.e., a model-free controller and a barrier-certification system, which discover RL-based control actions and sequentially filter them using a barrier certificate to satisfy operational constraints. Calculating a barrier function is generally challenging for a complex power system. This is addressed by representing the barrier function using neural networks (NNs) and data-based approaches. An adaptive method is introduced to certify the neural barrier function that perseveres barrier conditions, which is more compatible with online implementation. The proposed framework synthesizes a stabilizing controller that satisfies predefined safety regions. The effectiveness of the proposed framework is demonstrated via several comparative case studies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint

We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.

data-driven control↗
DOE OSTI2025

Safe and Robust Binary Classification and Fault Detection Using Reinforcement Learning

In this paper, we propose a learning-based method utilizing the Soft Actor-Critic (SAC) algorithm to train a binary Support Vector Machine (SVM) classifier. This classifier is designed to identify valid input spaces in high-dimensional, highly constrained systems while minimizing the total runtime of offline simulations. The simulations adapt their runtime based on the likelihood that a given training input will be informative to the classifier. Furthermore, we introduce a method for using the trained SAC model to predict whether a desired system input is likely to violate constraints, along with a technique to adjust the input as necessary. Additionally, we explore the potential of this model to detect faults or adversarial attacks within the system. The effectiveness of our approach is demonstrated through various simulations of challenging classification problems and a constrained quadrotor model.

Netter, Josh [Georgia Institute of Technology, Atl↗
DOE OSTI2024

Pedestrian-Involved Traffic Signal Optimization Using Decentralized Graph-based Multi-Agent Reinforcement Learning

Incorporating pedestrian movements in traffic signal timing optimization is essential for ensuring smooth and safe urban transportation systems. To develop an effective signal timing plan, it is crucial to comprehend how various pedestrian accommodation strategies impact both vehicle and pedestrian flow. This study explores the application of the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) method for signal timing optimization. It considers both pedestrian and vehicle traffic state and assesses the effects of fixed and adaptive pedestrian request response on traffic and signal timing. The evaluation of DGMARL encompasses vehicles Eco_PI which is fuel consumption impact related to vehicle stops and stop delay, and pedestrian waiting time to ensure overall system efficiency. The proposed approach was evaluated using a Digital Twin microscopic traffic simulation model of MLK Smart Corridor in Chattanooga, Tennessee. The evaluation outcomes of vehicles Eco_PI, pedestrian waiting time and serving time are compared with the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall to determine the most suitable approaches for various traffic conditions. The results indicated that, on average, the strategy of Signal timing optimization with adaptive pedestrian request response improved the Eco_PI by 32.43%, and Signal timing optimization with both automated pedestrian traffic demand and adaptive pedestrian request response improved the Eco_PI by 31.62% compared to the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall.

K Kumarasamy, Vijayalakshmi↗