Search NASA⌕ Search

SEARCH · Search NASA

Results for “Multi-agent control”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

A Secure Learning Control Strategy via Dynamic Camouflaging for Unknown Dynamical Systems under Attacks

This paper presents a secure reinforcement learning (RL) based control method for unknown linear time-invariant cyber-physical systems (CPSs) that are subjected to compositional attacks such as eavesdropping and covert attack. We consider the attack scenario where the attacker learns about the dynamic model during the exploration phase of the learning conducted by the designer to learn a linear quadratic regulator (LQR), and thereafter, use such information to conduct a covert attack on the dynamic system, which we refer to as doubly learning-based control and attack (DLCA) framework. We propose a dynamic camouflaging based attack-resilient reinforcement learning (ARRL) algorithm which can learn the desired optimal controller for the dynamic system, and at the same time, can inject sufficient misinformation in the estimation of system dynamics by the attacker. The algorithm is accompanied by theoretical guarantees and extensive numerical experiments on a consensus multi-agent system and on a benchmark power grid model.

Mukherjee, Sayak↗

Deep Reinforcement Learning Enabled Physical-Model-Free Two-Timescale Voltage Control Method for Active Distribution Systems

Active distribution networks are being challenged by frequent and rapid voltage violations due to renewable energy integration. Conventional model-based voltage control methods rely on accurate parameters of the distribution networks, which are difficult to achieve in practice. This paper proposes a novel physical-model-free two-timescale voltage control framework for active distribution systems. To achieve fast control of PV inverters, the whole network is first partitioned into several subnetworks using voltage-reactive power sensitivity. Then, the scheduling of PV inverters in the multiple sub-networks is formulated as Markov games and solved by a multi-agent soft actor-critic (MASAC) algorithm, where each subnetwork is modeled as an intelligent agent. All agents are trained in a centralized manner to learn a coordinated strategy while being executed based on only local information for fast response. For the slower time-scale control, OLTCs and switched capacitors are coordinated by a single agent-based SAC algorithm using the global information with considering control behaviors of the inverters. Particularly, the two-level agents are trained concurrently with information exchange according to the reward signal calculated from the data-driven surrogate model. Comparative tests with different benchmark methods on IEEE 33-and 123-bus systems and 342-node low voltage distribution system demonstrate that the proposed method can effectively mitigate the fast voltage violations and achieve systematical coordination of different voltage regulation assets without the knowledge of accurate system model.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Multi-Agent Reinforcement Learning for Distribution System Critical Load Restoration

Grid resilience has become a critical topic recently because of the increasing occurrence of extreme events and the growing integration of intermittent renewable energy sources. To build a resilient distribution system, this paper develops a multiagent reinforcement learning-based (MARL) method to coordinate distribution energy resources (DERs) dispatch, load pickup, and network reconfiguration for load restoration after a system outage. With the help of two types of control agents, namely critical load restoration (CLR) and coordination (COR) agents, system loads can be restored efficiently, given available resources. The effectiveness and superiority of the proposed algorithm are demonstrated through simulations and comparative studies on a real distribution feeder in Western Colorado.

distribution system↗

A Distributed Reinforcement Learning Yaw Control Approach for Wind Farm Energy Capture Maximization: Preprint

In this paper, we present a reinforcement-learning based distributed approach to wind farm energy capture maximization using yaw control, also known as wake steering. In order to maximize the power output of a wind farm, it is often necessary for individual turbines to decrease their own power output through yaw misalignment so as to deflect their wakes away from downstream turbines. Although using model-based methods to achieve yaw misalignment is one option, a model-free method might be better suited to incorporate factors that are difficult to model or changing conditions. We propose an algorithm that adapts concepts of temporal difference reinforcement learning distributed to a multi-agent environment that allows individual turbines to act so as to optimize overall wind farm output and react to unforeseen disturbances.

controls↗

Multi-Agent Safe Policy Learning for Power Management of Networked Microgrids

This paper presents a supervised multi-agent safe policy learning (SMAS-PL) method for optimal power management of networked microgrids (MGs) in distribution systems. While unconstrained reinforcement learning (RL) algorithms are black-box decision models that could fail to satisfy grid operational constraints, our proposed method considers AC power flow equations and other operational limits. Accordingly, the training process employs the gradient information of operational constraints to ensure that the optimal control policy functions generate safe and feasible decisions. Furthermore, we have developed a distributed consensus-based optimization approach to train the agents’ policy functions while maintaining MGs’ privacy and data ownership boundaries. After training, the learned optimal policy functions can be safely used by the MGs to dispatch their local resources, without the need to solve a complex optimization problem from scratch. Lastly, numerical experiments have been devised to verify the performance of the proposed method.

, policy gradient↗

Solving a system of linear equations: From centralized to distributed methods

For a wide range of control engineering applications, the problem of solving a system of linear equations is often encountered and has been well studied. Traditionally, this problem has been mainly solved in a centralized manner. However, for applications related to large-scale complex networked systems, centralized algorithms are often subjected to some practical issues due to limited computational power and communication bandwidth. As a promising and viable alternative, distributed algorithms can effectively address the issues associated with centralized algorithms by solving the problem efficiently in a multi-agent setting that accords with the distributed nature of networked systems. Distributed algorithms decompose the entire problem into many sub-problems that are solved by individual agents in a cooperative manner. In this survey paper, we provide a detailed overview of the state of the art relevant to distributed algorithms for solving a system of linear equations. We will first review basic distributed algorithms including both discrete-time and continuous-time algorithms. Then we will discuss the extended algorithms to achieve communication efficiency. Furthermore, we will also introduce distributed algorithms to obtain the minimum-norm solution for a system of linear equations with multiple solutions, as well as the least-squares solutions when there is no solution. Finally, the relationship of distributed algorithms for solving a system of linear equations to the existing distributed optimization algorithms is discussed.

Wang, Peng↗

Scalable Approaches to Selecting Key Entities in Large Networked Infrastructure Systems

This work aims at bringing advances in discrete optimization algorithms to solving practical engineering problems at scale. Often times, in many engineering design problems, there is a need to select a small set of influential or representative elements from a large ground set of entities in an optimal fashion. Submodular optimization provides for a formal way to solve such problems. Common examples with infrastructure systems involve sensor placement and identification of key entities with certain objectives. However, scaling these approaches to large infrastructure systems can be challenging because of the high computational complexity of the overall framework that include the optimization algorithms as well as high-complexity compute-oracles that provide the necessary objective function values. In this work, we explore a well-studied and widely-applicable paradigm, namely leader-selection in a multi-agent networked setting in the context of scalable methodologies. We demonstrate novel frameworks that utilize variations of accelerated submodular optimization algorithms along with linear-algebraic methods that can help accelerate the oracle computations. We further explore this combination in conjunction with graph partitioning paradigms to take advantage of the accelerated algorithms in a distributed setting. Finally we demonstrate the key findings on a practical problem in an operational setting. For this, we leverage an example road network with approximately 18k nodes and 27k edges in a traffic control application, where we seek a limited number of k=200 key intersections. This problem can be solved in a serial setting in just under 5 hours providing more than 2 orders of magnitude speed-up over methods that do not consider acceleration techniques.

Visweswara Sathanur, Arun↗

Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27% in a 24-hour and an average of 29.88% in a PM-peak-hour scenario.

Multi-Agent Reinforcement Learning, Digital Twin, ↗

Optimal CO 2 storage management considering safety constraints in multi-stakeholder multi-site GCS projects: A Markov game perspective

Geological carbon storage (GCS) projects could involve a diverse array of stakeholders or players from public, private, and regulatory sectors, each with different objectives and responsibilities. Given the complexity, scale, and long-term nature of GCS operations, determining whether individual stakeholders can independently optimize their interests — or whether collaborative coalition agreements are needed — remains a central question for effective GCS project planning and management. To access large, high-quality storage resources, future GCS deployment may increasingly occur in geologically connected sites, where shared geological features such as pressure space and reservoir pore capacity can lead to competitive behavior among stakeholders. In this work, we propose a paradigm based on Markov games to quantitatively investigate how different coalition structures affect the goals of stakeholders. We frame this multi-stakeholder multi-site problem as a multi-agent reinforcement learning problem with safety constraints. Our approach enables agents to learn optimal strategies while complying with safety regulations. We present an example where multiple operators are injecting CO 2 into their respective project areas in a geologically connected basin. To address the high computational cost of repeated simulations of high fidelity models, a previously developed surrogate model based on the Embed-to-Control (E2C) framework is employed. Our results demonstrate the effectiveness of the proposed framework in addressing optimal management of CO 2 storage when multiple stakeholders with different objectives and goals are involved.

58 GEOSCIENCES↗

Buyers Collusion in Incentivized Forwarding Networks: A Multi-Agent Reinforcement Learning Study

We present the issue of monetarily incentivized forwarding in a multi-hop mesh network architecture from an economic perspective. It is anticipated that credit-incentivized forwarding and relaying will be a simple method of exchanging transmission power and spectrum for connectivity. However, gateways and forwarding nodes, like any other free market, may create an oligopolistic market for the users they serve. In this study, a coalition scheme between buyers aims to address price control by gateways or nodes closer to gateways. In a Stackelberg competition game, buyer agents (users) and sellers (gateways) make decisions using reinforcement learning (RL), with decentralized Deep Q-Networks to buy and sell forwarding resources. We allow communication links between the buyers with a limited messaging space, without defining a collusion mechanism. The idea is to demonstrate that through messaging, and RL tacit collusion can emerge between agents in a decentralized setup. The multi-agent reinforcement learning (MARL) system is presented and analyzed from a machine-learning perspective. Moreover, MARL dynamics are discussed via mean field analysis to better understand divergence causes and make implementation recommendations for such systems. Finally, the simulation results show the results of coordination among the users.

42 ENGINEERING↗

Resilient Design of Continuous-time Distributed Optimization Algorithm in the Presence of Cyber-attacks

This paper presents a continuous-time resilient distributed optimization algorithm based on competitive interaction design method on connected graphs in the presence of adversaries. Here, the competitive interaction method allows us to design a network that protects the multi-agent systems from adversaries without requiring high network connectivity. In addition, the proposed algorithm does not require the global information about the number of adversaries. First, we show that the proposed distributed algorithm solves the resilient distributed optimization problem with no attack on the communication links. Second, we show that the proposed continuous-time distributed optimization algorithm on connected graphs converges to the small neighborhood of the optimal solution in the presence of cyber-attacks onto the communication channel. Simulations are presented to illustrate our theoretical results.

97 MATHEMATICS AND COMPUTING↗

Blockchain Enabled Intelligence of Federated Systems (BELIEFS): An attack-tolerant trustable distributed intelligence paradigm

In this article, a Blockchain Enabled Intelligence of Federated Systems (BELIEFS) is proposed to conduct cooperative control for the multi-regional large-scale power system with a multi-agents system (MAS). By establishing a two levels blockchain, each regional AI agent can simultaneously manage intra-regional controllers and cooperate with other AI agents. Under the consensus mechanism, the agents, which respectively conducted distributed deep reinforcement learning (DDRL) algorithm in multi-regions, can have the tolerant capability of malicious attacks in their training process. The demonstration of the proposed approach is within a multi-regional large-scale interconnected power system. Under the mode of "centralized dispatching and hierarchical management", this article aims to definite a mathematical model to deal with the control problem of the power systems. With the comparison experiments, the effectiveness and efficiency of our proposed method in the training process are verified. In addition, malicious attacks are set on the main chain and shard chains to verify the attack-tolerant capability. We expect that such approach and results can suggest a new paradigm of attack-tolerant trustable distributed AI deployment.

97 MATHEMATICS AND COMPUTING↗

Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning

This paper presents a novel federated reinforcement learning (Fed-RL) methodology to inject sufficient resiliency into the operations of the network of microgrids. We consider adversarial actions to the voltage and power control loop reference signals at the grid forming (GFM) inverters in the microgrids which are essential to integrate renewable resources. Therefore, we formulate a resilient reinforcement learning training setup that uses these adversarial injections to generate episodic trajectories and train the RL agents to alleviate their impact on performance. To circumvent the concerns about data-sharing and privacy for different owners of the microgrids in the networked setting, we bring in the aspects of the federated operation to propose novel Fed-RL algorithms. As the dynamics of each microgrid are coupled due to electrical interlinks, the conventional federated RL approaches using decoupled independent environments are not applicable, which leads us to propose a multi-agent vertically federated variation of actor-critic algorithms, namely federated soft actor-critic (FedSAC). We have performed numerical simulations on an IEEE 123-bus benchmark test feeder with three microgrids by creating a customized simulation setup by encapsulating the microgrid dynamic simulations in GridLAB-D/HELICS co-simulation platform with the OpenAI Gym environment and validated the proposed resilient and secured learning methodology.

Artificial Intelligence (AI), reinforcement learni↗

Learning Sequential Distribution System Restoration via Graph-Reinforcement Learning

We report a distribution service restoration algorithm as a fundamental resilient paradigm for system operators provides an optimally coordinated, resilient solution to enhance the restoration performance. The restoration problem is formulated to coordinate distribution generators and controllable switches optimally. A model-based control scheme is usually designed to solve this problem, relying on a precise model and resulting in low scalability. To tackle these limitations, this work proposes a graph-reinforcement learning framework for the restoration problem. We link the power system topology with a graph convolutional network, which captures the complex mechanism of network restoration in power networks and understands the mutual interactions among controllable devices. Latent features over graphical power networks produced by graph convolutional layers are exploited to learn the control policy for network restoration using deep reinforcement learning. The solution scalability is guaranteed by modeling distributed generators as agents in a multi-agent environment and a proper pre-training paradigm. Comparative studies on IEEE 123-node and 8500-node test systems demonstrate the performance of the proposed solution.

24 POWER TRANSMISSION AND DISTRIBUTION↗

ReLIC: Full-Scale Realization of Reinforcement Learning for Infrastructure Control

Prior efforts have shown that deep reinforcement learning (DRL) may provide a new method for controlling networked power systems. Though successful, prior approaches have not yet demonstrated their behavior on systems of realistic scale. This effort examined multiple theoretical and technical approaches to allow a DRL model to operate over a system of 2,000 buses or more. We find that allowing the DRL models to run training episodes in parallel provides near limitless efficiency gains, allowing us to train successful agents to behave on our Kuramoto transmission model of up to 4,000 buses. We further show that we can expand our PowerWorld DRL implementation to systems of up to 25 buses but struggle to go beyond this limit due to PowerWorld’s inability to run multiple instances at once. Finally, we examine a multi-agent approach and find that it performs as well if not better than our existing centralized approach.

97 MATHEMATICS AND COMPUTING↗

Autonomous Intelligent Charging/Discharging of Electric Vehicles using Distributed Multi-Agent ADMM Framework for Grid Ancillary Services

The increasing popularity of Electric Vehicles (EVs) in the distribution grid along with technological advancement in EV electronics such as vehicle to grid (V2G) technique has enabled them to participate in grid ancillary services. To achieve this, the EVs need to establish a contract with third-party aggregators and connect to a charging unit, either residential or commercial. At any time they are connected, the EVs can decide to take part in the ancillary services program offered to them by the aggregators. If agreed, the aggregators will use the EVs as a power source capable of charging/discharging power according to the input signal, and in return, they will be compensated. This inter-temporal nature of charging/discharging is also transforming the traditional optimal power flow (OPF) problem into a dynamic OPF problem. This chapter aims at developing a multi-layer time-dependent optimization algorithm to utilize EV potential and provide ancillary services while maximizing its utilization function. Specifically, in the upper layer, an autonomous distributed ADMM algorithm is developed to optimize the cost for charging/discharging EVs while using them to regulate the voltage at each bus in the distribution grid. The distributed ADMM algorithm is also expanded to the lower layer where the individual EVs active and reactive power is controlled for voltage regulation while maintaining the desired state of the charge of the vehicle at the end of the charging period. Here, the effectiveness and performance improvement of the proposed multi-layer algorithm is illustrated through analytical analysis and simulation results.

Rahman, Towfiq↗

Autonomous Intelligent Charging/Discharging of Electric Vehicles using Distributed Multi-Agent ADMM Framework for Grid Ancillary Services

The increasing popularity of Electric Vehicles (EVs) in the distribution grid along with technological advancement in EV electronics such as vehicle to grid (V2G) technique has enabled them to participate in grid ancillary services. To achieve this, the EVs need to establish a contract with third-party aggregators and connect to a charging unit, either residential or commercial. At any time they are connected, the EVs can decide to take part in the ancillary services program offered to them by the aggregators. If agreed, the aggregators will use the EVs as a power source capable of charging/discharging power according to the input signal, and in return, they will be compensated. This inter-temporal nature of charging/discharging is also transforming the traditional optimal power flow (OPF) problem into a dynamic OPF problem. This chapter aims at developing a multi-layer time-dependent optimization algorithm to utilize EV potential and provide ancillary services while maximizing its utilization function. Specifically, in the upper layer, an autonomous distributed ADMM algorithm is developed to optimize the cost for charging/discharging EVs while using them to regulate the voltage at each bus in the distribution grid. The distributed ADMM algorithm is also expanded to the lower layer where the individual EVs active and reactive power is controlled for voltage regulation while maintaining the desired state of the charge of the vehicle at the end of the charging period. The effectiveness and performance improvement of the proposed multi-layer algorithm is illustrated through analytical analysis and simulation results.

Rahman, Towfiq↗

A Distributed Model Identification Algorithm for Multi-Agent Systems: Preprint

In this study, we investigate agent-based approach for system model identification with emphasis on power distribution system applications. Departing from conventional practices of relying on historical data for offline model identification, we adopt online update approach utilizing real-time data by employing the latest data points for gradient computation. This methodology offers advantages including a large reduction in the communication network's bandwidth requirements by minimizing the data exchanged at each iteration and enabling the model to adapt in real-time to disturbances. Furthermore, we extend our model identification process from linear frameworks to more complex non-linear convex models. This extension is validated through numerical studies demonstrating improved control performance for a synthetic IEEE test case.

data-driven control↗