Search NASA⌕ Search

SEARCH · Search NASA

Results for “Operator learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,441 records · Page 80
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

FY23 ION Based Approaches to Address Labor and Knowledge Retention

This study will seek to outline the current problem facing the nuclear industry related to staffing, training, and retaining the necessary workforce for long-term sustainability. A review of published reports and knowledge gained by working in the nuclear energy sector will form the basis for this scoping study. This study will not seek to draw any conclusions regarding the underlying causes of these personnel issues but will attempt to ask questions and help outline future research. The Integrated Operations for Nuclear business model provides opportunities to partially mitigate some of the current issues facing staffing the nuclear fleet. Upgrading safety and control systems from analog to digital as advocated by the Integrated Operations for Nuclear process allows the nuclear facility to compete for the best and brightest candidates and offers them a broad and sustainable career path where their skills can be valued and utilized in the broader industrial sectors. This study will seek to identify the primary reasons that the nuclear industry is facing this labor crisis and seek to identify possible solutions that will be fully evaluated in future research.

42 ENGINEERING↗
DOE OSTI2023

Hierarchical Flexibility Offering Strategy for Integrated Hybrid Resources in Real-time Energy Markets

This paper proposes a hierarchical model for determining the energy flexibility offering strategy of integrated hybrid resources (IHRs) in power distribution systems to participate in real-time energy markets. The proposed model utilizes the scalability, fast response time, and uncertainty observation of deep reinforcement learning (DRL) to overcome the scalability issue of operating numerous flexible resources and deliverability of energy flexibility to the real-time markets in the presence of the network constraints. To that end, the power distribution system is divided into multiple IHRs, where different types of flexible loads, energy storage systems, and solar plants with controllable inverters are operated through local IHR controllers, trained by deep deterministic policy gradient (DDPG) algorithm. Active power request and reactive power capacity of IHRs are then transmitted to a central flexibility controller, where a quadratic optimization model ensures the deliverability of the energy flexibility to the real-time energy market by satisfying the distribution network constraints. The proposed model is implemented on the 123-bus test power distribution system, demonstrating the capability of DRL-based hierarchical model for scalable operation of IHRs in order to offer deliverable energy flexibility to the real-time energy market.

Majidi, Majid↗
DOE OSTI2020

Moving the Mission Forward with Renewable Energy

Presentation gives an overview of proven examples of renewable energy technologies. Learn how to properly utilize renewable energy technologies; research, select, fund, and deploy renewable technologies; best operate and maintain the federal government’s fleet of on-site solar photovoltaic (PV) systems; and learn best practices to detect and address performance issues with a PV system.

distributed energy↗
NASA NTRS2022

Making or Breaking a Rover: System Engineering Parameters On-Board the Mars 2020 Perseverance Rover

On February 18, 2021, Perseverance, NASA’s Jet Propulsion Laboratory’s (JPL’s) Mars 2020 Rover, successfully landed on Mars with all systems nominal, despite the risk surrounding the over 200,000 internal flight parameters that had to be properly configured. The Perseverance team defines these parameters as software variables that are configurable, commandable and retrievable from Earth. In 2015, the Mars 2020 project leaders focused on improving systems engineering of parameters based on their experiences from parameter management on previous Mars rovers (Curiosity, Opportunity, Spirit, and Pathfinder) and parameter failures of past missions, such as the mission-ending parameter of the Mars Climate Orbiter. The new rigorous development process allowed for efficient certification and effective implementation of the parameters, allowing the rover to approach and land on the red planet (the most challenging phase of the mission) with zero parameter issues. Although successful, the Perseverance team learned many lessons for how to better manage parameters for the continued surface operations of the Mars 2020 mission and future missions. This paper will discuss eight parameter-management topics for the Perseverance Mission. The first is parameter definition: how we define parameters on our mission, where they are physically located on the vehicle, and why we have so many of them. The second topic is the updated parameter flight software module from Curiosity, including details on the 99% reduction in parameter commands, new bulk configuration capabilities, and improved parameter traceability. The third topic is parameter selection for different mission phases; this includes improving and tweaking our preferred parameter settings until they become certification candidates and managing parameter configurations based on test venue throughout the mission life cycle. The fourth topic is our flight certification process; this includes certification of flight values for four different epochs in the mission: Launch, Entry Decent and Landing (EDL) - 6days, Landing + 5 Sols (Martian Days, still on Cruise Flight Software), and once are on Surface Flight Software (FSW). The fifth topic covers in-flight command implementation, along with details on testing, validation, and verification of those commands. In the sixth section, we will explain our use of open-source management tools, including how we used GitHub for version control and management approvals. The seventh topic will describe the ground tools used in operations, including capabilities of the in-house built tool called Parasol. The eighth and final topic will dig into lessons learned for improving parameter management in the future of this mission and others.

Roth, Brian↗
NASA NTRS2022

Making or Breaking a Rover- Systems Engineering Parameters On-Board the Mars 2020 Perseverance Rover

On February 18, 2021, Perseverance, NASA’s Jet Propulsion Laboratory’s (JPL’s) Mars 2020 Rover, successfully landed on Mars with all systems nominal, despite the risk surrounding the over 200,000 internal flight parameters that had to be properly configured. The Perseverance team defines these parameters as software variables that are configurable, commandable and retrievable from Earth. In 2015, the Mars 2020 project leaders focused on improving systems engineering of parameters based on their experiences from parameter management on previous Mars rovers (Curiosity, Opportunity, Spirit, and Pathfinder) and parameter failures of past missions, such as the mission-ending parameter of the Mars Climate Orbiter. The new rigorous development process allowed for efficient certification and effective implementation of the parameters, allowing the rover to approach and land on the red planet (the most challenging phase of the mission) with zero parameter issues. Although successful, the Perseverance team learned many lessons for how to better manage parameters for the continued surface operations of the Mars 2020 mission and future missions. This paper will discuss eight parameter-management topics for the Perseverance Mission. The first is parameter definition: how we define parameters on our mission, where they are physically located on the vehicle, and why we have so many of them. The second topic is the updated parameter flight software module from Curiosity, including details on the 99% reduction in parameter commands, new bulk configuration capabilities, and improved parameter traceability. The third topic is parameter selection for different mission phases; this includes improving and tweaking our preferred parameter settings until they become certification candidates and managing parameter configurations based on test venue throughout the mission life cycle. The fourth topic is our flight certification process; this includes certification of flight values for four different epochs in the mission: Launch, Entry Decent and Landing (EDL) - 6days, Landing + 5 Sols (Martian Days, still on Cruise Flight Software), and once are on Surface Flight Software (FSW). The fifth topic covers in-flight command implementation, along with details on testing, validation, and verification of those commands. In the sixth section, we will explain our use of open-source management tools, including how we used GitHub for version control and management approvals. The seventh topic will describe the ground tools used in operations, including capabilities of the in-house built tool called Parasol. The eighth and final topic will dig into lessons learned for improving parameter management in the future of this mission and others.

Roth, Brian↗
NASA NTRS2024

Safe Exploration: Sensorimotor Assessments for Early Extravehicular Activities

BACKGROUND: Artemis missions will require a new level of crew autonomy around periods of gravitational transition, where sensorimotor disturbances are at their highest. There is a need to define performance thresholds for key sensorimotor assessments that indicate when performance in early extravehicular activities (EVAs) might be impacted or unsafe. This panel presentation will discuss the development of sensorimotor assessments for determining crew preparedness of early EVAs by utilizing a novel portable sensorimotor disorientation analog and other spaceflight analogs. OVERVIEW: To define performance thresholds, a proposed set of sensorimotor assessment tasks must be validated under various spaceflight analogs. A Sensorimotor Disorientation Analog (SDA) was developed that could induce varying levels of disorientation through combined vestibular (galvanic vestibular stimulation (GVS)) and proprioceptive (weighted chest, ankles, and wrists) disruptions. The SDA was first pilot tested using subjective feedback from previously flown astronauts to determine the levels of disorientation that mimic motor performance immediately (R+0) and +24 hours (R+1) postflight. A second study was performed using healthy non-astronaut ground subjects to validate the SDA levels by comparing to astronaut postflight data. The validated SDA was utilized in a third study to map performance in the proposed set of sensorimotor assessment tasks to operational tasks. The assessment tasks were defined based on lessons learned from Apollo and subject matter experts (e.g., flight surgeons) to include the following: 1) aid in progressive adaptation to the novel gravitational environment; 2) provide opportunities to develop strategies to recover from off-nominal body positions; and 3) mimic operational tasks such that crew can self-assess their potential ability to complete their missions. This presentation will conclude with a discussion on future validation studies of the proposed assessment tasks using other spaceflight analogs such as centrifugation and gravity offload systems. DISCUSSION: Exploration class missions will require crew to be able to self-assess and treat their sensorimotor dysfunction after gravity transitions, and in off-nominal situations they may be required to perform provocative, challenging tasks soon after landing. This panel presentation will discuss current and ongoing research strategies to address the sensorimotor risk on safe exploration during Artemis missions.

Sarah Moudy↗
DOE OSTI2022

A Scalable Method for Decarbonizing Modular Building Solutions: Preprint

The decarbonization movement emphasizes the shift in focus from energy efficiency to directly reducing global-warming impact. Blokable, LLC, a vertically integrated modular builder with an all-electric portfolio, worked with NREL on a roadmap to decarbonize its high-performance building product at a relative cost advantage by utilizing the learning curves of mass production. Previous decarbonization literature focused on either i) lifecycle assessments, or ii) efficiency measures. These decarbonization exercises were bespoke to individual building projects and did not consider positive feedback loops of builder experience or process repetition. Vertically integrated, prefab builders possess the unique ability to leverage learning and repetition to decarbonize their design-build-operate process. This collaboration between Blokable and NREL resulted in a decarbonization strategy utilizing the company's scaling and production efficiencies, on-site renewable energy and storage, and the projected evolution of building components over time based on trends and emerging legislation. The method developed here encompasses a growing business model, lifecycle carbon assessment, and projected changes in product and grid emissions over time due to existing trends and emerging legislation. This methodology incorporates learning-curve efficiencies gleaned from scaled manufacturing, as well as open-source tools integration for energy and carbon accounting. The output projects and compares cost and carbon savings per modular unit as production increases to 10,000 dwelling units annually over 15 years. The resulting roadmap illustrates a path to roughly 60% carbon savings and beyond-net-zero-energy performance at no incremental cost by 2030. The methodology can be mapped to other integrated or productized builders for methodical decarbonization.

affordable housing↗
DOE OSTI2022

A Scalable Method for Decarbonizing Modular Building Solutions

The decarbonization movement emphasizes the shift in focus from energy efficiency to directly reducing global-warming impact. Blokable, LLC, a vertically integrated modular builder with an all-electric portfolio, worked with NREL on a roadmap to decarbonize its high-performance building product at a relative cost advantage by utilizing the learning curves of mass production. Previous decarbonization literature focused on either i) lifecycle assessments, or ii) efficiency measures. These decarbonization exercises were bespoke to individual building projects and did not consider positive feedback loops of builder experience or process repetition. Vertically integrated, prefab builders possess the unique ability to leverage learning and repetition to decarbonize their design-build-operate process. This collaboration between Blokable and NREL resulted in a decarbonization strategy utilizing the company's scaling and production efficiencies, on-site renewable energy and storage, and the projected evolution of building components over time based on trends and emerging legislation. The method developed here encompasses a growing business model, lifecycle carbon assessment, and projected changes in product and grid emissions over time due to existing trends and emerging legislation. This methodology incorporates learning-curve efficiencies gleaned from scaled manufacturing, as well as open-source tools integration for energy and carbon accounting. The output projects and compares cost and carbon savings per modular unit as production increases to 10,000 dwelling units annually over 15 years. The resulting roadmap illustrates a path to roughly 60% carbon savings and beyond-net-zero-energy performance at no incremental cost by 2030. The methodology can be mapped to other integrated or productized builders for methodical decarbonization.

affordable housing↗
DOE OSTI2024

ION Work Reduction Opportunity Realization Demonstration

The purpose of this research was to realize one of the advanced training work reduction opportunities first presented in the Idaho National Laboratory (INL) report, “Process for Significant Nuclear Work Function Innovation Based on Integrated Operations Concepts” (INL/EXT-21-64134) [1], with a nuclear power plant (NPP) research partner. Researchers modernized two trainings: (1) an accredited instructor-led training (ILT) overview course on Westinghouse DS 480-volt (V) circuit breakers to a multimedia-focused computer-based-training (CBT) learning module, and (2) an on-demand chaptered video on how to properly rack and un-rack a Westinghouse DS 480-V circuit breaker. These modernized work products were developed and implemented in a manner consistent with the industry guidelines found in Institution of Nuclear Power Operations (INPO) Teaching and Learning 23-001 [2]. Researchers calculated that the modernized accredited training course reduced the time necessary to prepare and deliver the training material by a factor of 8:1. The amount of time learners spend in class could be reduced by this same factor. In other words, if a course took 8 hours to deliver a class, the new CBT instruction would take just over 1 hour. The researchers noted that the requirement for any practicum training by the learners with the instructor(s) would remain in place. But through interviews with new and experienced learners, the researchers discovered that the confidence of these learners in performing the racking and un-racking of the circuit breaker improved as a result of using the new modernized CBT process. Additionally, the learners who tested the modernized work products enjoyed the modernized CBT and the learning video significantly more than current in-class learning methods. These are encouraging results for the nuclear industry, as this modernization of training can be applied to other classes and is scalable across the industry. In line with the Integrated Operations for Nuclear (ION) model, positive workload analysis supports the investment of resources in modernizing NPP training processes and infrastructure. Implementation of the advanced training technologies in this report is likely to result in substantive long-term workload benefits to instructors and learners and result in hard-dollar savings on contractor spends. Additionally, investment in these modernized training processes will result in improved learner proficiency. The results of this research can be applied to additional operator, technical, and general training topics to provide additional workload and learning benefits in addition to what was explored.

42 ENGINEERING↗
DOE OSTI2020

Considerations for using Privacy Preserving Machine Learning Techniques for Safeguards

In international nuclear safeguards, the International Atomic Energy Agency (IAEA) is tasked with inspecting and verifying nuclear facilities and their activities. Data analytics and machine learning to support inspections require large amounts of data that nuclear facility operators may consider proprietary or sensitive, so the IAEA may not have full access. Allowing computation over private data without compromising its security therefore has value for safeguards inspections and analysis. Privacy-preserving machine learning (PPML) consists of security-focused techniques that allow data analytics and machine learning algorithms to run on sensitive data without revealing it. This includes ideas like homomorphic encryption (HE), secure multiparty computation (SMPC), and secure enclaves. HE allows algorithms and mathematical operations to be conducted directly on the encrypted data instead of first decrypting it. With SMPC, multiple entities collaboratively compute over distributed data such that no party is able to directly view any others’ original data. Secure enclaves allow computation to take place in a separate and heavily blocked-off section of a CPU. Techniques like these allow for several potential use cases in which the security of data is essential. With SMPC, machine learning models can be trained over the input data from multiple entities, resulting in a model that all users can benefit from without leaking the input data from any particular entity. With SMPC or a zero-knowledge proof (ZKP), an algorithm returning some single answer or truth value can be run on someone else’s data without ever needing to see that data, potentially allowing for verification or proof of some underlying question. HE can allow for outsourcing computation on data to a hostile or untrusted environment. Although most of the research in this field resides within the health and financial domains, tools from PPML may have similar applications in nuclear safeguards. Allowing the IAEA to compute over proprietary information, such as process models and raw sensor data using PPML techniques, provides the baseline for running complex analytics without needing direct unencrypted access to the underlying data, maintaining its privacy. Important limitations to consider for these techniques include the efficiency and level of security required. The security of HE and SMPC come at the cost of speed—the significant amount of overhead means that algorithms implemented in these protocols and encryption schemes are slower than when run on plaintext. Additionally, several important parameters determine what techniques or protocols are used based on the security requirements. SMPC protocols may need to be selected for resistance against a party that attempts to deviate from the protocol to distort the result or gain access to additional information, and a protocol secure against these attacks may further increase the overhead of the algorithm.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2014

Learning the Task Management Space of an Aircraft Approach Model

Validating models of airspace operations is a particular challenge. These models are often aimed at finding and exploring safety violations, and aim to be accurate representations of real-world behavior. However, the rules governing the behavior are quite complex: nonlinear physics, operational modes, human behavior, and stochastic environmental concerns all determine the responses of the system. In this paper, we present a study on aircraft runway approaches as modeled in Georgia Tech's Work Models that Compute (WMC) simulation. We use a new learner, Genetic-Active Learning for Search-Based Software Engineering (GALE) to discover the Pareto frontiers defined by cognitive structures. These cognitive structures organize the prioritization and assignment of tasks of each pilot during approaches. We discuss the benefits of our approach, and also discuss future work necessary to enable uncertainty quantification.

Validation↗
DOE OSTI2020

Deep Reinforcement Learning for Residential HVAC Control with Consideration of Human Occupancy

The Artificial Intelligence (AI) development described herein uses model-free Deep Reinforcement Learning (DRL) to minimize energy cost during residential heating, ventilation, and air conditioning (HVAC) operation. Building cooling loads and HVAC operation are difficult to accurately model due to complexity, lack of measurements and data, and model specific performance, so online machine learning is used to allow for real-time readjustment in performance. Energy costs for the multi-zone cooling unit shown in this work are minimized by scheduling on/off commands around dynamic prices. By taking advantage of precooling events that take place when the price is low, the agent is able to reduce operational cost without violating user comfort. The DRL controller was tested in simulation where the learner achieved a 43.89% cost reduction when compared to traditional, fixed-setpoint operation. The system is now ready for the next phase of testing in a live, real-time home environment.

Mckee, Evan↗
NASA NTRSDate not supplied

Initial Development of A Digital Twin Model for an Electrified Aircraft Propulsion Emulation Rig

In support of aviation fuel burn and emission reduction goals, NASA is pursing high-payoff research investments that promise to transform aviation. This includes investments in Electrified Aircraft Propulsion (EAP), which relies on the generation, storage, transmission, and use of electrical power for producing thrust and optimizing propulsion system efficiency. Multiple technology challenges must be addressed to unlock the full potential of EAP. This includes advances in propulsion controls, which will be vital for ensuring coordinated efficient operation of the complex integrated subsystems that comprise EAP architectures. To support EAP controls research, the NASA Glenn Research Center has developed the Hybrid Propulsion Emulation Rig (HyPER). The HyPER laboratory hardware includes shaft-mounted electric machines, power converters, power supplies, power distribution cables, and an energy storage device that can be reconfigured to represent a variety of EAP architectures. It also includes an integrated real-time computer system that hosts developed EAP control software and turbomachinery simulations. This enables the electrical system and rotating shafts of EAP designs to be implemented in actual hardware and integrated with turbomachinery simulations and system-level EAP control logic implemented in software. In this form, the HyPER laboratory provides a partially simulated, partially hardware-in-the-loop test environment enabling the initial development and evaluation of EAP control technology. A prerequisite for the development of EAP control designs is the availability of a system model that accurately reflects the operation of the electrical system hardware. To support this need, a digital twin model of the HyPER electrical system hardware is under development. This model is being coded in the MATLAB Simulink environment and uses the NASA-developed Electrical Modeling and Thermal Analysis Toolbox (EMTAT) to construct a digital twin framework. EMTAT contains generic electrical component building blocks that are simulated at turbomachinery timescales. Associated inputs and outputs allow the blocks to be combined to model complete electrical systems. The EMTAT blocks also contain adjustable internal maps and parameters that can be set to reflect the operation of a specific electrical component. For the HyPER digital twin, the settings of these EMTAT block internal maps and parameters is determined through machine learning approaches applied to characterization run data collected from the laboratory. During characterization runs the laboratory electrical system hardware is subjected to a full range of torque, speed, and power settings. Acquired data is then used to estimate EMTAT block parameters using a variety of machine learning techniques. The resulting digital twin model is found to match the operation of actual HyPER hardware with an accuracy suitable for control development purposes. It also holds promise for other applications including modeling the performance of HyPER laboratory reconfigurations and model-based anomaly detection. Planned follow-on work to automate post-processing of acquired laboratory data to update the HyPER digital twin model will also be presented and discussed.

Electrified Aircraft Propulsion↗
DOE OSTI2022

Multivariate Time Series Anomaly Detection with Few Positive Samples

Given the scarcity of anomalies in real-world applications, the majority of literature has been focusing on modeling normality. The learned representations enable anomaly detection as the normality model is trained to capture certain key underlying data regularities under normal circumstances. In practical settings, particularly industrial time series anomaly detection, we often encounter situations where a large amount of normal operation data is available along with a small number of anomaly events collected over time. This practical situation calls for methodologies to leverage these small number of anomaly events to create a better anomaly detector. In this paper, we introduce two methodologies to address the needs of this practical situation and compared them with recently developed state of the art techniques. Our proposed methods anchor on representative learning of normal operation with autoregressive (AR) model along with loss components to encourage representations that separate normal versus few positive examples. We applied the proposed methods to two industrial anomaly detection datasets and demonstrated effective performance in comparison with approaches from literature. Our study also points out additional challenges with adopting such methods in practical applications.

Xue, Feng↗
DOE OSTI2021

Machine Intelligence to Detect, Characterise, and Defend against Influence Operations in the Information Environment

Social media has enabled a new era of manipulation in the information and cognitive domains. Deceptive content—misleading, falsified, and fabricated—is routinely created and spread in the modern social media environment with the intent to create confusion and widen political and social divides, and exploit the societal conflict exacerbated by these divides in the real-world (aka physical domain). Such disinformation campaigns demonstrate a threat to the integrity of economic, political, cultural, public health, and national security institutions around the world. In this work we overview our artificial intelligence (AI) capabilities to detect, describe, and defend against information operations on Twitter as an example social platform to understand the influence of misleading and falsified content diffusion and better enable those charged with defending against such manipulation to enable responsive parties to counter it. We first present novel linguistically-informed deep learning (DL) models for misinformation and disinformation detection, and present an in-depth linguistic analysis of psycho-linguistic markers across broad deception categories. We then demonstrate how our models perform in the multilingual and multimodal setting and categorize falsified and misleading content based on the intent to deceive. We also provide a large-scale analysis to describe user behavior and spread patterns while engaging with deceptive content and report novel findings about the immediate diffusion of deceptive content by characterizing the vulnerable sub-populations and their demographics, and explicitly measuring speed and scale of deception spread to uncover who shares deceptive content, how quickly, how much, and how evenly. In addition, we measure audience reactions to misinformation and disinformation at scale, distinguishing the reactions of users identified as bots versus humans. Finally, we take advantage of deep translation and generation models to create unique solutions for real-time defense against digital deception and discuss how to apply causal inference to prescribe and intervene into strategic communications jointly across information, cognitive, and physical domains.

artificial intelligence, deep learning, neural lan↗
DOE OSTI2026

Short-term electricity load forecasting: Application-driven evaluation of machine learning models across spatial and temporal scales

As we transition towards a decarbonized economy, the integration of variable renewable energy resources and new demands (e.g., electric vehicles, heat pumps) into the electricity grid places unprecedented pressure on grid operators to effectively anticipate and manage peak load. In this context, machine learning algorithms are proving to be indispensable for accurate short-term load forecasting, a crucial task to address these challenges. This study benchmarks 6 machine learning algorithms, including three neural networks and three tree-based algorithms, across various levels of spatial aggregation and time horizons (1, 4, 8, 24, and 48 h). The central contribution of this work is the comparison and analysis of load forecasting models not only based on statistical metrics, but also based on a novel error metric, which evaluates the cost implications of forecast errors for power system stakeholders. Results show that tree-based models outperform neural networks, based on statistical metrics, and yield less skewed error distributions for most spatial scales. However, through the lens of the novel error metric, neural networks are the more competitive choice, especially for forecast horizons that exceed 8 h. The study concludes with actionable recommendations to grid operators and highlights the need for the development of error metrics that link forecasting accuracy to operational costs. To promote transparency and open science, the datasets and Python code are open-sourced via a supplementary repository.

Houben, Nikolaus↗
DOE OSTI2023

Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning

This paper presents a novel federated reinforcement learning (Fed-RL) methodology to inject sufficient resiliency into the operations of the network of microgrids. We consider adversarial actions to the voltage and power control loop reference signals at the grid forming (GFM) inverters in the microgrids which are essential to integrate renewable resources. Therefore, we formulate a resilient reinforcement learning training setup that uses these adversarial injections to generate episodic trajectories and train the RL agents to alleviate their impact on performance. To circumvent the concerns about data-sharing and privacy for different owners of the microgrids in the networked setting, we bring in the aspects of the federated operation to propose novel Fed-RL algorithms. As the dynamics of each microgrid are coupled due to electrical interlinks, the conventional federated RL approaches using decoupled independent environments are not applicable, which leads us to propose a multi-agent vertically federated variation of actor-critic algorithms, namely federated soft actor-critic (FedSAC). We have performed numerical simulations on an IEEE 123-bus benchmark test feeder with three microgrids by creating a customized simulation setup by encapsulating the microgrid dynamic simulations in GridLAB-D/HELICS co-simulation platform with the OpenAI Gym environment and validated the proposed resilient and secured learning methodology.

Artificial Intelligence (AI), reinforcement learni↗
DOE OSTI2022

Deep Reinforcement Learning From Demonstrations to Assist Service Restoration in Islanded Microgrids

Microgrids can be operated in island mode during utility grid outages to support service restoration and improve system resilience. To schedule and dispatch distributed energy resources (DERs) in an islanded microgrid, conventional model-based methods rely on accurate distribution network models and lack generalization and adaptability. Data-driven methods are promising for DER coordination but face practical challenges such as potential hazards to microgrids during online training and insufficient online training opportunities due to low outage rates. This paper presents a novel two-stage learning framework that builds on the deep deterministic policy gradient from demonstrations to identify an optimal restoration strategy. At the pre-training stage, imitation learning is applied to equip the control agent with expert experiences to guarantee acceptable initial performance. At the online training stage, action clipping, reward shaping, and expert demonstrations are leveraged to ensure safe exploration while accelerating the training process. In conclusion, the proposed method is illustrated using the IEEE 123-node system and compared with a representative model-based method and the standard deep deterministic policy gradient method to prove solution accuracy and demonstrate increased computational efficiency.

Du, Yan↗