SEARCH · Search NASA
Results for “MARL”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
One-Dimensional (1D) Reactive Transport with PFLOTRAN: Marl Leaching Case - Progress Status.
Abstract not provided.
One-Dimensional (1D) Reactive Transport with PFLOTRAN: Marl Leaching Case - Progress Status.
Abstract not provided.
Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint
We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.
Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning
Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.
PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.
A Cooperative Multi-Agent Deep Reinforcement Learning Framework for Real-Time Residential Load Scheduling
Internet-of-Things (IoT) enabled monitoring and control capabilities are enabling increasing numbers of household users with controllable loads to actively participate in smart grid energy management. Realizing an efficient real-time energy management system that takes advantage of these developments requires novel techniques for managing the increased complexity of the control action space in resolving multiple challenges such as the uncertainty in energy prices and renewable energy output along with the need to satisy physical grid constraints such as transformer capacity. Addressing these challenges, we develop a multi-household energy management framework for residential units connected to the same transformer and containing DERs such as PV, ESS and controllable loads. The goal of our framework is to schedule controllable household appliances and ESS such that the cost of procuring electricity from the utility over a horizon is minimized while physical grid constraints are satisfied at each scheduling step. Traditional energy management frameworks either perform global optimization to satisfy grid constraints but suffer from high computational complexity (for example Integer Program, Mixed Integer Programming frameworks and centralized reinforcement learning) or perform decentralized real-time energy management without satisfying global grid constraints (for example multi-agent reinforcement learning with no cooperation). In contrast, we propose a cooperative multiagent reinforcement learning (MARL) framework that i) operates in real-time, and ii) performs explicit collaboration to satisfy global grid constraints. The novelty in our framework is two fold. Firstly, our framework trains multiple independent learners (IL) for each household in parallel using historical data and performs real-time inferencing of control actions using the most recent system state. Secondly, our framework contains a low complexity knapsack based cooperation agent which combines the outputs of ILs to minimize cost while satisfying grid constraints. Simulation results show that our cooperative MARL approach achieves significant cost improvement over centralized reinforcement learning and day-ahead planning baselines. Moreover, our approach strictly satisfies physical constraints with no apriori knowledge of system dynamics while the baseline approaches have occasional violations. We also measure the training and inference time by ranging the number of households from 1 to 25. Results show that our cooperative MARL approach scales best among various approaches.
Enhancing Autonomous Control of Microreactors Using Multi-Agent Reinforcement Learning
In order for microreactors to be economically competitive, operation costs will need to be minimized through some degree of autonomous control. Previous work has demonstrated the effectiveness of reinforcement learning (RL) for load-following control in a drum-controlled microreactor. This study extends that work by exploring the potential of RL to independently control each of the reactor’s drums. We compare a single-agent RL approach with a multi-agent RL (MARL) framework, testing them for generalization across different load-following power profiles and control timescales, and for robustness in cases of randomly disabled control drums. Since the point kinetics simulation environment used in this study cannot resolve spatial effects, we assume that in the absence of spatially localized disturbances, optimal drum movements should be symmetrical. We demonstrate that single-agent RL is able to achieve accurate performance only when symmetric actions are ignored; otherwise, it fails to train a useful controller. Meanwhile, the MARL framework performs symmetric actions by design and trains a robust, accurate agent, as evidenced by mean absolute errors in power matching of 0.41% for the training power profile, 0.68% for a profile with half the drums disabled, and 0.21% for a profile on a realistic load-following time horizon.
Nuclear microreactor transient and load-following control with deep reinforcement learning
The economic feasibility of nuclear microreactors will depend on minimizing operating costs through advancements in autonomous control, especially when these microreactors are operating alongside other types of energy systems (e.g., renewable energy). This study explores the application of deep reinforcement learning (RL) for real-time drum control in microreactors, exploring performance in regard to load-following scenarios. By leveraging a point kinetics model with thermal and xenon feedback, we first establish a baseline using a single-output RL agent, then compare it against a traditional proportional–integral–derivative (PID) controller. This study demonstrates that RL controllers, including both single- and multi-agent RL (MARL) frameworks, can achieve similar or even superior load-following performance as traditional PID control across a range of load-following scenarios. In short transients, the RL agent was able to reduce the tracking error rate in comparison to PID by one half to one third. Over extended 300-minute load-following scenarios in which xenon feedback becomes a dominant factor, PID maintained better accuracy, but RL still remained within a 1% error margin despite being trained only on short-duration scenarios. This highlights RL’s strong ability to generalize and extrapolate to longer, more complex transients, affording substantial reductions in training costs and reduced overfitting. Furthermore, when control was extended to multiple drums, MARL enabled independent drum control as well as maintained reactor symmetry constraints without sacrificing performance---an objective that standard single-agent RL could not learn. We also found that, as increasing levels of Gaussian noise were added to the power measurements, the RL controllers were able to maintain lower error rates than PID, and to do so with at least 10% and upwards of 150% less control effort. These findings illustrate RL's potential for autonomous nuclear reactor control, laying the groundwork for future integration into high-fidelity simulations and experimental validation efforts.
Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning
In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationary. Learning non-stationary policies is challenging and typically requires sophisticated or inefficient algorithms. Motivated by the prevalence of this control problem in real-world complex systems, we introduce a simple framework for learning non-stationary policies for multi-timescale MARL. Our approach uses available information about agent timescales to define and learn periodic multi-agent policies. In detail, we theoretically demonstrate that the effects of non-stationarity introduced by multiple timescales can be learned by a periodic multi-agent policy. To learn such policies, we propose a policy gradient algorithm that parameterizes the actor and critic with phase-functioned neural networks, which provide an inductive bias for periodicity. The framework's ability to effectively learn multi-timescale policies is validated on a gridworld and building energy management environment.
Deep Multi-Agent Reinforcement Learning for Real-World Signalized Traffic Corridor Control
Signalized traffic control problem has been addressed recently with deep Reinforcement Learning (RL) approaches involving diverse state, action, and reward structures. While significant progress has been noted in the literature, open challenges still remain in the areas of adaptive signal phase timing, coordination in a multi-intersection corridor setting, and consideration of real-world traffic conditions. In the context of deep RL-based problem framing, extensions are needed that enable adaptive signal phase timings in an intersection agent's action space, computationally efficient information sharing among neighboring signalized intersection agents along a corridor, and experimentation in realistic simulation environments. In this paper, we develop a deep Advantage Actor Critic (A2C) multi-agent RL (MARL) approach capturing the research extensions above and apply it within a real-world calibrated Aimsun Next traffic corridor simulation model based on traffic data from the City of Coral Gables, Florida. For a multi-intersection corridor control setting, our numerical simulation experiments with a decentralized A2C MARL algorithm applied at different time periods led to a total average corridor travel delay reduction (expressed in seconds/mile averaged over vehicles) from 4.9% to 19.9% compared to state-of-the-art actuated control.
Buyers Collusion in Incentivized Forwarding Networks: A Multi-Agent Reinforcement Learning Study
We present the issue of monetarily incentivized forwarding in a multi-hop mesh network architecture from an economic perspective. It is anticipated that credit-incentivized forwarding and relaying will be a simple method of exchanging transmission power and spectrum for connectivity. However, gateways and forwarding nodes, like any other free market, may create an oligopolistic market for the users they serve. In this study, a coalition scheme between buyers aims to address price control by gateways or nodes closer to gateways. In a Stackelberg competition game, buyer agents (users) and sellers (gateways) make decisions using reinforcement learning (RL), with decentralized Deep Q-Networks to buy and sell forwarding resources. We allow communication links between the buyers with a limited messaging space, without defining a collusion mechanism. The idea is to demonstrate that through messaging, and RL tacit collusion can emerge between agents in a decentralized setup. The multi-agent reinforcement learning (MARL) system is presented and analyzed from a machine-learning perspective. Moreover, MARL dynamics are discussed via mean field analysis to better understand divergence causes and make implementation recommendations for such systems. Finally, the simulation results show the results of coordination among the users.
Intensified inundation shifts a freshwater wetland from a CO 2 sink to a source
Abstract Climate change has altered global precipitation patterns and has led to greater variation in hydrological conditions. Wetlands are important globally for their soil carbon storage. Given that wetland carbon processes are primarily driven by hydrology, a comprehensive understanding of the effect of inundation is needed. In this study, we evaluated the effect of water level (WL) and inundation duration (ID) on carbon dioxide (CO 2 ) fluxes by analysing a 10‐year (2008–2017) eddy covariance dataset from a seasonally inundated freshwater marl prairie in the Everglades National Park. Both gross primary production (GPP) and ecosystem respiration (ER) rates showed declines under inundation. While GPP rates decreased almost linearly as WL and ID increased, ER rates were less responsive to WL increase beyond 30 cm and extended inundation periods. The unequal responses between GPP and ER caused a weaker net ecosystem CO 2 sink strength as inundation intensity increased. Eventually, the ecosystem tended to become a net CO 2 source on a daily basis when either WL exceeded 46 cm or inundation lasted longer than 7 months. Particularly, with an extended period of high‐WLs in 2016 (i.e., WL remained >40 cm for >9 months), the ecosystem became a CO 2 source, as opposed to being a sink or neutral for CO 2 in other years. Furthermore, the extreme inundation in 2016 was followed by a 4‐month postinundation period with lower net ecosystem CO 2 uptake compared to other years. Given that inundation plays a key role in controlling ecosystem CO 2 balance, we suggest that a future with more intensive inundation caused by climate change or water management activities can weaken the CO 2 sink strength of the Everglades freshwater marl prairies and similar wetlands globally, creating a positive feedback to climate change.
PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems: Preprint
We present the PowerGridworld software package to provide users with a light-weight, modular, and customizable framework for creating power systems-focused, multi-agent gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). While many frameworks exist for training multi-agent (MA) RL policies, none exist to rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multi-device) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an open-source software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning multi-agent RL policies using both OpenAI's MADDPG and RLLib's PPO algorithms where, in both cases, at least some subset of agents incorporate elements of the power flow solution at each time step as part of their reward (negative cost) structures.
Decentralized Voltage Control with Peer-to-peer Energy Trading in a Distribution Network
Utilizing distributed renewable and energy storage resources via peer-to-peer (P2P) energy trading has long been touted as a solution to improve energy system’s resilience and sustainability. Consumers and prosumers (those who have energy generation resources), however, do not have expertise to engage in repeated P2P trading, and the zero-marginal costs of renewables present challenges in determining fair market prices. To address these issues, we propose a multi-agent reinforcement learning (MARL) framework to help automate consumers’ bidding and management of their solar PV and energy storage resources, under a specific P2P clearing mechanism that utilizes the so-called supply-demand ratio. In addition, we show how the MARL framework can integrate physical network constraints to realize decentralized voltage control, hence ensuring physical feasibility of the P2P energy trading and paving ways for real-world implementations.
Integrating Aquatic Metabolism and Net Ecosystem CO2 Balance in Short- and Long-Hydroperiod Subtropical Freshwater Wetlands
Abstract How aquatic primary productivity influences the carbon (C) sequestering capacity of wetlands is uncertain. We evaluated the magnitude and variability in aquatic C dynamics and compared them to net ecosystem CO 2 exchange (NEE) and ecosystem respiration ( R eco ) rates within calcareous freshwater wetlands in Everglades National Park. We continuously recorded 30-min measurements of dissolved oxygen (DO), water level, water temperature ( T water ), and photosynthetically active radiation (PAR). These measurements were coupled with ecosystem CO 2 fluxes over 5 years (2012–2016) in a long-hydroperiod peat-rich, freshwater marsh and a short-hydroperiod, freshwater marl prairie. Daily net aquatic primary productivity (NAPP) rates indicated both wetlands were generally net heterotrophic. Gross aquatic primary productivity (GAPP) ranged from 0 to − 6.3 g C m −2 day −1 and aquatic respiration ( R Aq ) from 0 to 6.13 g C m −2 day −1 . Nonlinear interactions between water level, T water , and GAPP and R Aq resulted in high variability in NAPP that contributed to NEE. Net aquatic primary productivity accounted for 4–5% of the deviance explained in NEE rates. With respect to the flux magnitude, daily NAPP was a greater proportion of daily NEE at the long-hydroperiod site (mean = 95%) compared to the short-hydroperiod site (mean = 64%). Although we have confirmed the significant contribution of NAPP to NEE in both long- and short-hydroperiod freshwater wetlands, the decoupling of the aquatic and ecosystem fluxes could largely depend on emergent vegetation, the carbonate cycle, and the lateral C flux.
Space geodetic monitoring of engineered structures: The ongoing destabilization of the Mosul dam, Iraq
We present a detailed survey of the ongoing destabilization process of the Mosul dam. The dam is located on the Tigris river and is the biggest hydraulic structure in Iraq. From a geological point of view the dam foundation is poor due to a site geology formed by alternating strata of highly soluble materials including gypsum, anhydrite, marl and limestone. Here we present the first multi-sensor cumulative deformation map for the dam generated from space-based interferometric synthetic aperture radar measurements from the Italian constellation COSMO-SkyMed and the European sensor Sentinel-1a over the period 2014–2016 that we compare to an older dataset spanning 2004–2010 acquired with the European Envisat satellite. We found that deformation was rapid during 2004–2010, slowed in 2012–2014 and increased since August 2014 when grouting operations stopped due to the temporary capture of the dam by the self proclaimed Islamic State. We model the inferred deformation using a Markov chain Monte Carlo approach to solve for change in volume for simple tensile dislocations. Results from recent and historical geodetic datasets suggests that the volume dissolution rate remains constant when the equivalent volume of total concrete injected during re-grouting operations is included in the calculations.
The ongoing destabilization of the mosul dam as observed by synthetic aperture radar interferometry
We present a detailed survey here on the ongoing destabilization process of the Mosul dam. The dam is located on the Tigris river and is the biggest hydraulic structure in Iraq. From a geological point of view the dam foundation is unstable due to the underlying geology that is formed by alternate and variable strata of highly soluble materials such as gypsum, anhydrite, marl and limestone. We present the first comprehensive multi-sensor cumulative deformation map for the dam generated from space-based synthetic aperture radar (SAR) measurements from the Italian constellation COSMO-SkyMed and the European Sentinel-1a satellite. We compared 2014-2016 data to an historic dataset spanning 2004-2010 acquired with the Envisat ASAR sensor. We found that deformation was rapid during 2004-2010, slowed down in 2012-2014, and restarted in August 2014 when grouting operations stopped due to the temporary capture of the dam by the self proclaimed Islamic State in Iraq and Syria (ISIS). We took advantage of the availability of data from multiple SAR satellites to infer the deformation at the dam in great spatial and temporal detail and shed new light on the processes of the ongoing destabilization. This study highlights how new constellations of SAR sensors together with the availability of historical datasets are leading to important advances in deformation monitoring of small scale geologic and manmade features.