Search NASA⌕ Search

Engineering topics

Ni, Zhen

Publications and source records attributed to Ni, Zhen.

A Modified Maximum Entropy Inverse Reinforcement Learning Approach for Microgrid Energy Scheduling

Increasing popularity of integrating distributed energy resources (DERs) into the power system brings a challenge to optimize the microgrid dispatch policy. The reinforcement learning methods suffer from a long-time problem with the theoretical assumption of the objective/reward function for the microgrid system. Although the traditional inverse reinforcement learning (IRL) approaches can solve this problem to some extent, they encounter a limitation of complex computations for state visitation frequency in the large and continuous state space. To alleviate this limitation, we propose a modified maximum entropy IRL (MMIRL) method to extract the reward function from the expert demonstrations for solving the microgrid energy scheduling problem. The proposed MMIRL algorithm is promising in recovering the reward function and learning the dispatch policy compared to conventional approaches. Case studies are performed in an energy arbitrage problem and a microgrid system with DERs. Results substantiate that the proposed MMIRL approach can learn the dispatch policy with more than 99% efficiency and outperforms other comparative methods.

artificial intelligence, reinforcement learning, m↗

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

24 POWER TRANSMISSION AND DISTRIBUTION↗

An Efficient Distributed Reinforcement Learning for Enhanced Multi-Microgrid Management

Economic dispatch in multi-microgrid (MMG) systems requires coordinating distributed energy resources (DERs) of different microgrids, which leads to a significant increase in the number of states for energy management. In these cases, traditional reinforcement learning (RL) approaches become computationally expensive or output a solution that causes extra-operating costs for the system. This paper proposes an RL approach that employs local learning agents to interact with microgrid environments in a distributed manner and aggregates the outcomes to train the global agent to learn the policy for the MMG system. This distributed exploration and aggregation process provides an effective solution and guides the global agent to learn the dispatch policy efficiently. Case studies are performed on a system with three microgrids with different types of DERs. Results obtained using the proposed RL and comparisons with conventional methods substantiate the effectiveness of the proposed approach in terms of operation costs, computation time, and peak-to-average ratio.

Das, Avijit↗

Overvoltage Prevention and Curtailment Reduction Using Adaptive Droop-Based Supplementary Control in Smart Inverters

Recent developments in the renewable energy sector have seen an unprecedented growth in residential photovoltaic (PV) installations. However, high PV penetration levels often lead to overvoltage problems in low-voltage (LV) distribution feeders. Smart inverter control such as active power curtailment (APC)-based overvoltage control can be implemented to overcome these challenges. The APC technique utilizes a constant droop-based approach which curtails power rigidly, which can lead to significant energy curtailment in the LV distribution feeders. In this paper, different variations of the APC technique with linear, quadratic, and exponential droops have been analyzed from the point-of-view of energy curtailment for a LV distribution network in North America. Further, a combinatorial approach using various droop-based APC methods in conjunction with adaptive dynamic programming (ADP) as a supplementary control scheme has also been proposed. The proposed approach minimizes energy curtailment in the LV distribution network by adjusting the droop gains. Simulation results depict that ADP in conjunction with exponential droop reduces the energy curtailment to approximately 50% compared to using the standard linear droop.

42 ENGINEERING↗

Q-Learning Based Impact Assessment of Propagating Extreme Weather on Distribution Grids: Preprint

Increasing number of power outage events due to extreme weather condition is hampering us socioeconomically. Preparing in advance for the extreme weather event is critical and can help utility operators to reduce grid damages, restore grid service quickly, allocate energy resources and repair crews strategically, and hence dramatically increase grid resilience. In this paper, we propose a method to identify the sequence of worst impact zones in the power grid caused by extreme weather events based on Q-learning (a reinforcement learning algorithm). To quantify weather severity and it’s effect on the grid, we model the impact of extreme weather on the grid as a function of intensity, vulnerability and exposure. A modified IEEE 123-node distribution feeder is presented in a mesh grid and experimented for sequences of zones identification. Finally, simulation results present the identified sequences and their associated impacts on the grid caused by the extreme weather events.

distribution system↗

Q-Learning-Based Impact Assessment of Propagating Extreme Weather on Distribution Grids

The increasing number of power outage events due to extreme weather conditions is hampering us socioeconomically. Preparing in advance for the extreme weather event is critical. It can help utility operators to reduce grid damages, restore grid service quickly, allocate energy resources and repair crews strategically, and hence dramatically increase grid resilience. In this paper, we propose a method to identify the sequence of worst impact zones in the power grid caused by extreme weather events based on Q-learning (a reinforcement learning algorithm). To quantity the weather severity and its effect on the grid, we model the impact of extreme weather on the grid as a function of intensity, vulnerability, and exposure. A modified IEEE 123-node distribution feeder is presented in a mesh grid and experimented for sequences of zones identification. Finally, simulation results present the identified sequences and their associated impacts on the grid caused by extreme weather events.

distribution system↗

Experimental Validation of Approximate Dynamic Programming Based Optimization and Convergence on Microgrid Applications

Stochastic optimization can better address uncertainties in power system problems. However, when state space and action space become large, many existing approaches become computationally expensive and even infeasible. Approximate dynamic programming (ADP) attracts researchers’ attention as a powerful tool for solving power system optimization problems with reduced computational cost. In this paper, in light of the existing literature, we investigate how the ADP approach with post-decision value function approximation converges to the nearly optimal solution with improved computational speed and experimentally validate the performance of the approach for a microgrid energy optimization problem. The approximation error versus the number of iteration is studied for convergence analysis of the post-decision ADP. A flowchart is provided to illustrate the proposed ADP algorithm for a microgrid energy optimization problem. The performance of ADP and dynamic programming (DP) is compared in terms of optimization error and computational time. It has found that the post-decision ADP approach can achieve competitive optimality with improved computational speed compared to the traditional DP.

Das, Avijit↗

An Analysis of Post Attack Impacts and Effects of Learning Parameters on Vulnerability Assessment of Power Grid

Due to the increasing number of heterogeneous devices connected to electric power grid, the attack surface increases the threat actors. Game theory and machine learning are being used to study the power system failures caused by external manipulation. Most of existing works in the literature focus on one-shot process of attacks and fail to show the dynamic evolution of the defense strategy. In this paper, we focus on an adversarial multistage sequential game between the adversaries of the smart electric power transmission and distribution system. We study the impact of exploration rate and convergence of the attack strategies (sequences of action that creates large scale blackout based on the system capacity) based on the reinforcement learning approach. We also illustrate how the learned attack actions disrupt the normal operation of the grid by creating transmission line outages, bus voltage violations, and generation loss. This simulation studies are conducted on IEEE 9 and 39 bus systems. The results show the improvement of the defense strategy through the learning process. The results also prove the feasibility of the learned attack actions by replicating the disturbances created in simulated power system.

attack impacts↗