Search NASA⌕ Search

SEARCH · Search NASA

Results for “multiagent systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A Multiagent Deep Reinforcement Learning-Enabled Dual-Branch Damping Controller for Multimode Oscillation

Here, this study develops a multiagent deep reinforcement learning (MADRL)-enabled framework for the decentralized cooperative control of a novel dual-branch (DB) damping controller for both low-frequency oscillation (LFO) and ultralow-frequency oscillation (ULFO). It has two branches, each of which consists of a proportional resonance (PR) and a second-order polynomial that is designed to handle target oscillation modes. To improve the robustness of the controller to system uncertainties, MADRL is developed, where multiagents are centrally trained to obtain the coordinated adaptive control policy while being executed in a decentralized manner to provide the optimal parameter setting for each controller with only local states. Comparisons with the IEEE 10-machine 39-bus system demonstrate that the proposed method achieves better robustness to uncertainties, lower communication delay, and single-point failure, as well as damping control performances for both LFO and ULFO.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Agent-Based Distributed Energy Resources for Supporting Intelligence at the Grid Edge

This article proposes a novel multi-agent framework that can link various forms of resources and power electronic systems into distributed energy resources (DER). The proposed multiagent architecture can also integrate DERs to a central controller for optimization and control to support the grid. To demonstrate the flexibility of this novel framework, the developed agent system is applied to a set of end-use systems. Furthermore, the agent framework is validated in hardware using controller-hardware-in-the-loop simulation platform.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Multi-agent AI collaboration for digital twin development and assessment

Developing a digital twin (DT) model involves different steps that encompass formulating requirements, model development, implementation, and assessment with respect to real applications. Human expertise is required to coordinate and implement different steps in the DT development and assessment process. However, certain parts of this process can be automated using artificial intelligence (AI) agents for efficient workflow development. In this work, we test and analyze a multiagent AI collaboration with humans in the loop to automate different elements of the DT development and assessment process. To implement the workflow for multiagent AI DT development and assessment, we use Autogen, a multiagent framework developed by Microsoft. Autogen offers a modular and flexible framework for configuring and designing task-specific multiagent workflows. In this framework, large language models (LLMs) form the core intelligence of the AI agents where the quality and performance of the automated element is governed by the inherent capabilities and knowledge base of the LLM. We use retrieval augmented generation to supplement the LLM with relevant domain-specific information for DT requirement formulation. We illustrate this multiagent workflow using a case study on a thermal energy storage system, focusing on how AI agents can collaborate with humans to expedite and optimize different elements of DT development and assessment process.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2021

Decentralized Voltage Control of Large-Scale Distribution System with PVs Based on MADRL

This paper proposes a model-free decentralized control framework for the voltage regulation of large-scale distribution systems through the coordinated control of PV inverters. This is achieved by developing a novel interaction mechanism between the surrogate model and the centralized training and decentralized execution multiagent deep reinforcement learning framework. Specifically, the sparse Gaussian processes regression method is first utilized to develop the surrogate model of the original distribution system for reward calculation during the training stage, where each agent represents a sub-region in the centralized fashion for coordination strategy learning. After that, the learned control rules are used to inform controllers within each sub-region for real-time decisions with only local measurements. Comparative tests among various methods on the EPRI Ckt5 test system demonstrate the effectiveness of the proposed method.

distribution system↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning

Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.

chance constraints↗
DOE OSTI2021

A coupled human–natural system analysis of freshwater security under climate and population change

Limited water availability, population growth, and climate change have resulted in freshwater crises in many countries. Jordan’s situation is emblematic, compounded by conflict-induced population shocks. Integrating knowledge across hydrology, climatology, agriculture, political science, geography, and economics, we present the Jordan Water Model, a nationwide coupled human–natural-engineered systems model that is used to evaluate Jordan’s freshwater security under climate and socioeconomic changes. The complex systems model simulates the trajectory of Jordan’s water system, representing dynamic interactions between a hierarchy of actors and the natural and engineered water environment. A multiagent modeling approach enables the quantification of impacts at the level of thousands of representative agents across sectors, allowing for the evaluation of both systemwide and distributional outcomes translated into a suite of water-security metrics (vulnerability, equity, shortage duration, and economic well-being). Model results indicate severe, potentially destabilizing, declines in freshwater security. Per capita water availability decreases by approximately 50% by the end of the century. Without intervening measures, >90% of the low-income household population experiences critical insecurity by the end of the century, receiving <40 L per capita per day. Widening disparity in freshwater use, lengthening shortage durations, and declining economic welfare are prevalent across narratives. To gain a foothold on its freshwater future, Jordan must enact a sweeping portfolio of ambitious interventions that include large-scale desalinization and comprehensive water sector reform, with model results revealing exponential improvements in water security through the coordination of supply- and demand-side measures.

42 ENGINEERING↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint

We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.

data-driven control↗
DOE OSTI2024

Quantum for Energy Systems and Technologies

Quantum Information Science (QIS) is expected to profoundly change the practice of science and engineering in the coming decades. QIS technology exploits quantum phenomena for performing tasks that are impossible to do today and is a rapidly progressing field, fueled by large investments from the private sector and governments. Its importance to the U.S. economy and national security is underscored by the National Quantum Initiative Act (NQIA) passed in December 2018, which creates a coordinated multiagency program to support research and training in QIS. After the NIQA signed into law, NETL has launched an initiative to apply QIS to problems encountered in energy technology development. In Jan. 2020, an Quantum for Energy Systems & Technologies (QUEST) working group was formed to establish a workforce capable of developing, reviewing, managing, and advising on QIS-related technologies for NETL and FECM. Since then, the QUEST team have been working on developing quantum sensing technology and performing quantum computing to solve energy-related problems. This poster summarized the QUEST team’s activities & accomplishments on QIS targeting energy-related applications.

Paudel, Hari P.↗
DOE OSTI2021

A Modular and Transferable Reinforcement Learning Framework for the Fleet Rebalancing Problem

Mobility on demand (MoD) systems show great promise in realizing flexible and efficient urban transportation. However, significant technical challenges arise from operational decision making associated with MoD vehicle dispatch and fleet rebalancing. For this reason, operators tend to employ simplified algorithms that have been demonstrated to work well in a particular setting. To help bridge the gap between novel and existing methods, we propose a modular framework for fleet rebalancing based on model-free reinforcement learning (RL) that can leverage an existing dispatch method to minimize system cost. In particular, by treating dispatch as part of the environment dynamics, a centralized agent can learn to intermittently direct the dispatcher to reposition free vehicles and mitigate against fleet imbalance. We formulate RL state and action spaces as distributions over a grid partitioning of the operating area, making the framework scalable and avoiding the complexities associated with multiagent RL. Numerical experiments, using real-world trip and network data, demonstrate that RL reduces waiting time by 28% to 38% for the same-day evaluation, 17% to 44% for cross-day evaluation, and 22% to 25% for cross-season evaluation compared with no rebalancing scenarios. This approach has several distinct advantages over baseline methods including: improved system cost; high degree of adaptability to the selected dispatch method; and the ability to perform scale-invariant transfer learning between problem instances with similar vehicle and request distributions.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2014

Assessment of Offshore Wind System Design, Safety, and Operation Standards

This report is a deliverable for a project sponsored by the U.S. Department of Energy (DOE) entitled National Offshore Wind Energy Resource and Design Data Campaign -- Analysis and Collaboration (contract number DE-EE0005372; prime contractor -- AWS Truepower). The project objective is to supplement, facilitate, and enhance ongoing multiagency efforts to develop an integrated national offshore wind energy data network. The results of this initiative are intended to 1) produce a comprehensive definition of relevant met-ocean resource assets and needs and design standards, and 2) provide a basis for recommendations for meeting offshore wind energy industry data and design certification requirements.

17 WIND ENERGY↗
DOE OSTI2019

A Cooperative Multi-Agent Deep Reinforcement Learning Framework for Real-Time Residential Load Scheduling

Internet-of-Things (IoT) enabled monitoring and control capabilities are enabling increasing numbers of household users with controllable loads to actively participate in smart grid energy management. Realizing an efficient real-time energy management system that takes advantage of these developments requires novel techniques for managing the increased complexity of the control action space in resolving multiple challenges such as the uncertainty in energy prices and renewable energy output along with the need to satisy physical grid constraints such as transformer capacity. Addressing these challenges, we develop a multi-household energy management framework for residential units connected to the same transformer and containing DERs such as PV, ESS and controllable loads. The goal of our framework is to schedule controllable household appliances and ESS such that the cost of procuring electricity from the utility over a horizon is minimized while physical grid constraints are satisfied at each scheduling step. Traditional energy management frameworks either perform global optimization to satisfy grid constraints but suffer from high computational complexity (for example Integer Program, Mixed Integer Programming frameworks and centralized reinforcement learning) or perform decentralized real-time energy management without satisfying global grid constraints (for example multi-agent reinforcement learning with no cooperation). In contrast, we propose a cooperative multiagent reinforcement learning (MARL) framework that i) operates in real-time, and ii) performs explicit collaboration to satisfy global grid constraints. The novelty in our framework is two fold. Firstly, our framework trains multiple independent learners (IL) for each household in parallel using historical data and performs real-time inferencing of control actions using the most recent system state. Secondly, our framework contains a low complexity knapsack based cooperation agent which combines the outputs of ILs to minimize cost while satisfying grid constraints. Simulation results show that our cooperative MARL approach achieves significant cost improvement over centralized reinforcement learning and day-ahead planning baselines. Moreover, our approach strictly satisfies physical constraints with no apriori knowledge of system dynamics while the baseline approaches have occasional violations. We also measure the training and inference time by ranging the number of households from 1 to 25. Results show that our cooperative MARL approach scales best among various approaches.

14 SOLAR ENERGY↗
DOE OSTI2023

Understanding and Predictability of Integrated Mountain Hydroclimate (Workshop Report)

Mountainous systems cover approximately 23% of Earth’s land and are distributed across all continents. They can capture and store atmospheric moisture that is then cycled through the terrestrial surface and subsurface system, released to downstream communities, and cycled back to the atmosphere. Mountain hydroclimate—characterized by steep gradients, geological, ecological, and biogeochemical diversity—is influenced by topographic forcing and elevated warming and susceptible to large subseasonal to multidecadal variability and rapid changes. Terrestrial hydrological and biogeochemical cycles also experience cascading effects from global warming impacts, such as multidecadal declines in mountain snowpack, longer growing seasons, and increased frequency and severity of extreme events like droughts and wildfires. However, little is known about the effects of these impacts and their feedbacks on climate systems and surface-subsurface compartments. Also unknown are the full implications of changing hydroclimate and extreme events on hydro biogeochemical cycles across atmosphere, terrestrial, and human systems in mountain regions and beyond. This knowledge gap is critical, given human reliance on mountain systems for stable water supply and quality. Mountain systems’ increasing vulnerability to climate change and human perturbations motivates the need to improve understanding of integrated mountain hydroclimate (IMHC) systems and their feedbacks and impacts on humans across scales. However, due to large heterogeneity and strong gradients, coupled natural-human processes in mountain regions present significant challenges for observations, modeling, predictions, and projections. Motivated by gaps in mountain hydroclimate understanding, observations, and modeling and the need for credible projections of future changes, the U.S. Department of Energy’s (DOE) Biological and Environmental Research (BER) program organized a virtual workshop on “Understanding and Predictability of Integrated Mountain Hydroclimate.” Sponsored by BER’s Earth and Environmental Systems Sciences Division (EESSD), the workshop aimed to inform and catalyze EESSD’s growing interests in enhancing predictive understanding of IMHC. Organizers structured the workshop to identify (1) knowledge gaps, (2) observational and modeling challenges, (3) short-term (1 to 3 years) and long-term (10 years and beyond) research opportunities, and (4) strategies for fostering collaboration and coordination. To address the outstanding challenges of IMHC, the workshop included two sessions organized by disciplinary, cross-disciplinary, and crosscutting science topics. The disciplinary and cross-disciplinary topics focused on essential IMHC elements: atmosphere, terrestrial, and human systems and their interactions. Breakout sessions on disciplinary and cross-disciplinary topics facilitated identification of crosscutting topics and central emerging themes. Session 1 focused on connecting existing DOE investments to accelerate progress related to scientific challenges in understanding mountain hydroclimate. In Session 2, participants further explored key Session 1 takeaways through the lens of multiagency collaborations and coordination.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Multirobot Collaborative Pursuit Target Robot by Improved MADDPG

Policy formulation is one of the main problems in multirobot systems, especially in multirobot pursuit-evasion scenarios, where both sparse rewards and random environment changes bring great difficulties to find better strategy. Existing multirobot decision-making methods mostly use environmental rewards to promote robots to complete the target task that cannot achieve good results. This paper proposes a multirobot pursuit method based on improved multiagent deep deterministic policy gradient (MADDPG), which solves the problem of sparse rewards in multirobot pursuit-evasion scenarios by combining the intrinsic reward and the external environment. The state similarity module based on the threshold constraint is as a part of the intrinsic reward signal output by the intrinsic curiosity module, which is used to balance overexploration and insufficient exploration, so that the agent can use the intrinsic reward more effectively to learn better strategies. The simulation experiment results show that the proposed method can improve the reward value of robots and the success rate of the pursuit task significantly. The intuitive change is obviously reflected in the real-time distance between the pursuer and the escapee, the pursuer using the improved algorithm for training can get closer to the escapee more quickly, and the average following distance also decreases.

Zhou, Xiao↗
DOE OSTI2026

Extreme-scale EV charging infrastructure planning for last-mile delivery using high-performance parallel computing

Here, this paper addresses stochastic charger location and allocation problems under queue congestion for last-mile delivery using electric vehicles (EVs). The objective is to decide where to open charging stations and how many chargers of each type to install, subject to budgetary and waiting-time constraints. We formulate the problem as a mixed-integer non-linear program, where each station-charger pair is modeled as a multiserver queue with stochastic arrivals and service times to capture the notion of waiting in fleet operations. The model is extremely large, with billions of variables and constraints for a typical metropolitan area; even loading the model in solver memory is difficult, let alone solving it. To address this challenge, we develop a Lagrangian-based dual decomposition framework that decomposes the problem by station and leverages parallelization on high-performance computing systems, where the subproblems are solved by using a cutting plane method and their solutions are collected at the master level. We also develop a three-step rounding heuristic to transform the fractional subproblem solutions into feasible integral solutions. Computational experiments on data from the Chicago metropolitan area with hundreds of thousands of households and thousands of candidate stations show that our approach produces high-quality solutions in cases where existing exact methods cannot even load the model in memory. We also analyze various policy scenarios, demonstrating that combining existing depots with newly built stations under multiagency collaboration substantially reduces costs and congestion. These findings offer a scalable and efficient framework for developing sustainable large-scale EV charging networks.

Capacity allocation↗
DOE OSTI2025

Controlled environment agriculture: An opportunity to strengthen interagency research collaboration in the US government

Challenges facing food production and agricultural systems are increasingly interconnected with economic, security, health, and equity issues, among others. Threats such as extreme weather, economic volatility, and shrinking water resources and arable land, influence our ability to maintain a safe and resilient food supply. One promising solution to these threats is controlled environment agriculture (CEA). In many cases, CEA can drastically reduce the amount of water and land used in crop production while increasing productivity. Operations may be established in nearly any environment and harvests can take place year-round, supporting food system resiliency and sustainability. CEA sits at the nexus of a number of disciplines and industries, making it well suited for transdisciplinary and multi-institutional research coordination. Herein, authors from multiple US government agencies present CEA as a case study in improving cross-agency research collaboration. The federal government houses a range of scientific expertise and research capabilities, positioning scientists to lead national and global efforts in transdisciplinary, interagency approaches to complex challenges. Navigating cross-agency collaboration can be a challenge, especially coordinating across different scientific disciplines, geographic locations, and funding mechanisms. To enhance multiagency efforts, collaborators could prioritize (i) organizing personnel and resources, (ii) enhancing existing multiagency collaborations, and (iii) focusing on further opportunities for coordination. Adopting these approaches could enable federal researchers to reinforce and advance academic and industry efforts to address current CEA challenges while solidifying the United States as a leader in this arena.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗