Search NASA⌕ Search

SEARCH · Search NASA

Results for “offline policy improvement”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

8 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Dynamic Role-Based Access Control Policy for Smart Grid Applications: An Offline Deep Reinforcement Learning Approach

Role-based access control (RBAC) is adopted in the information and communication technology domain for authentication purposes. However, due to a very large number of entities within organizational access control (AC) systems, static RBAC management can be inefficient, costly, and can lead to cybersecurity threats. In this paper, a novel hybrid RBAC model is proposed, based on the principles of offline deep reinforcement learning (RL) and Bayesian belief networks. The considered framework utilizes a fully offline RL agent, which models the behavioral history of users as a Bayesian belief-based trust indicator. Thus, the initial static RBAC policy is improved in a dynamic manner through off-policy learning while guaranteeing compliance of the internal users with the security rules of the system. By deploying our implementation within the smart grid domain and specifically within a Distributed Energy Resources (DER) ecosystem, we provide an end-to-end proof of concept of our model. Finally, detailed analysis and evaluation regarding the offline training phase of the RL agent are provided, while the online deployment of the hybrid RL-based RBAC model into the DER ecosystem highlights its key operation features and salient benefits over traditional RBAC models.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e

We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab). Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale. We treat the Mu2e accelerator system as a Markov Decision Process suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and enhance training stability. A key innovation in our approach is the integration of a neuralized Proportional-Integral-Derivative (PID) controller into the policy function, resulting in a significant improvement in the Spill Duty Factor (SDF) by 13.6%, surpassing the performance of the current PID controller baseline by an additional 1.6%. This paper presents the preliminary offline results based on a differentiable simulator of the Mu2e accelerator. It paves the groundwork for real-time implementations and applications, representing a crucial step towards automated proton beam intensity control for the Mu2e experiment.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

Reinforcement Learning for Intentional Islanding in Resilient Power Transmission Systems

Intentional islanding is the process of identifying and deliberately decomposing the transmission network to form self-sustained islands from an endangered network during disruptions to improve resilience and security. Most existing intentional islanding models are offline resilience decision tools and hence do not provide outage responses in a timely manner. In this paper, a reinforcement learning (RL) based model for intentional islanding is developed, which offers real-time switching control, online deployability, and adaptability to varying system conditions. The intentional islanding process is formulated as a Markov decision process, where the optimal transmission switching policy is learned using the RL approach. The control policy is learned over an environment that encompasses a Power System Simulator for Engineering (PSS/E) model of the transmission network, facilitated by an interface to the standard openAI Gym framework. The proposed RL-based methodology aims to form stable and self-sustainable islands by ensuring voltage stability while reducing the power mismatch in the formed islands. A proximal policy optimization algorithm is designed, which is suitable for controlling the on/off status of the switches with multi-layer perceptron as value and actor networks. The effectiveness of the proposed framework in the self-recovery of the grid by island formation is applied on the modified IEEE 39-bus test network and validated by dynamic simulations.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

In recent years, advanced control strategies based on Deep Reinforcement Learning (DRL) proved to be effective in optimizing the management of integrated energy systems in buildings, reducing energy costs and improving indoor comfort conditions when compared to traditional reactive controllers. However, the scalability and implementation of DRL controllers are still limited since they require a considerable amount of time before converging to a near-optimal solution. This issue is currently addressed in literature through the offline pre-training of the DRL agent. However this solution results in two main critical issues: (1) the need to develop a building surrogate model to perform the training task, and (2) the need to perform a fine-tuning process over several training episodes to obtain a near-optimal control policy. In this context, this paper introduces an Online Transfer Learning (OTL) strategy that exploits two knowledge-sharing techniques, weight-initialization and imitation learning, to transfer a DRL control policy from a source office building to various target buildings in a simulation environment coupling EnergyPlus and Python. A DRL controller based on discrete Soft Actor–Critic (SAC) is trained on the source building to manage the operation of a cooling system consisting of a chiller and a thermal storage. Several target buildings are defined to benchmark the performance of the OTL strategy with that of a Rule-Based Controller (RBC) and two DRL-based control strategies, deployed in offline and online fashion. The strategy adopted for OTL emulates the real world implementation with a simulation process by implementing the transferred DRL agent for a single episode in the target buildings. Target buildings have the same geometrical features and are served by the same energy system as the source building, but differ in terms of weather conditions, electricity price schedules, occupancy patterns, and building envelope efficiency levels. The results show that the OTL strategy can reduce the cumulated sum of temperature violations on average by 50% and 80% respectively when compared to RBC and online DRL while enhancing the energy system operation with electricity cost savings ranging between 20% and 40%. Furthermore, the OTL agent performs slightly worse than the offline DRL controller but it does not require any modeling effort and can be implemented directly on target buildings emulating a real-world implementation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Simulations of aerosol pH in China using WRF-Chem (v4.0): sensitivities of aerosol pH and its temporal variations during haze episodes

Aerosol pH is a fundamental property of aerosols in terms of atmospheric chemistry and its impact on air quality, climate, and health. Precise estimation of aerosol pH in chemical transport models (CTMs) is critical for aerosol modeling and thus influences policy development that partially relies on results from model simulations. We report the Weather Research and Forecasting Model coupled with Chemistry (WRF-Chem) simulated PM 2.5 pH over China during a period with heavy haze episodes in Beijing, and explore the sensitivity of the modeled aerosol pH to factors including emissions of nonvolatile cations (NVCs) and NH 3 , aerosol phase state assumption, and heterogeneous production of sulfate. We find that default WRF-Chem could predict spatial patterns of PM 2.5 pH over China similar to other CTMs, but with generally lower pH values, largely due to the underestimation of alkaline species (NVCs and NH 3 ) and the difference in thermodynamic treatments between different models. Increasing NH 3 emissions in the model would improve the modeled pH in comparison with offline thermodynamic model calculations of pH constrained by observations. In addition, we find that the aerosol phase state assumption and heterogeneous sulfate production are important in aerosol pH predictions for regions with low relative humidity (RH) and high anthropogenic SO 2 emissions, respectively. These factors should be better constrained in model simulations of aerosol pH in the future. Analysis of the modeled temporal trend of PM 2.5 pH in Beijing over a haze episode reveals a clear decrease in pH from 5.2 ± 0.9 in a clean period to 3.6 ± 0.5 in a heavily polluted period. The increased acidity under more polluted conditions is largely due to the formation and accumulation of secondary species including sulfuric acid and nitric acid, even though being modified by alkaline species (NVCs, NH 3 ). Our result suggests that NO 2 oxidation is unlikely to be important for heterogeneous sulfate production during the Beijing haze as the effective pH for NO 2 oxidation of S(IV) is at a higher pH of ~6.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A comparative study of two-way and offline coupled WRF v3.4 and CMAQ v5.0.2 over the contiguous US: performance evaluation and impacts of chemistry–meteorology feedbacks on air quality

The two-way coupled Weather Research and Forecasting and Community Multiscale Air Quality (WRF-CMAQ) model has been developed to more realistically represent the atmosphere by accounting for complex chemistry–meteorology feedbacks. In this study, we present a comparative analysis of two-way (with consideration of both aerosol direct and indirect effects) and offline coupled WRF v3.4 and CMAQ v5.0.2 over the contiguous US. Long-term (5 years from 2008 to 2012) simulations using WRF-CMAQ with both offline and two-way coupling modes are carried out with anthropogenic emissions based on multiple years of the U.S. National Emission Inventory and chemical initial and boundary conditions derived from an advanced Earth system model (i.e., a modified version of the Community Earth System Model/Community Atmospheric Model). The comprehensive model evaluations show that both two-way WRF-CMAQ and WRF-only simulations perform well for major meteorological variables such as temperature at 2 m, relative humidity at 2 m, wind speed at 10 m, precipitation (except for against the National Climatic Data Center data), and shortwave and longwave radiation. Both two-way and offline CMAQ also show good performance for ozone (O3) and fine particulate matter (PM 2.5 ). Due to the consideration of aerosol direct and indirect effects, two-way WRF-CMAQ shows improved performance over offline coupled WRF and CMAQ in terms of spatiotemporal distributions and statistics, especially for radiation, cloud forcing, O 3 , sulfate, nitrate, ammonium, elemental carbon, tropospheric O 3 residual, and column nitrogen dioxide (NO 2 ). For example, the mean biases have been reduced by more than 10 W m –2 for shortwave radiation and cloud radiative forcing and by more than 2 ppb for max 8 h O 3 . However, relatively large biases still exist for cloud predictions, some PM 2.5 species, and PM 10 that warrant follow-up studies to better understand those issues. The impacts of chemistry–meteorological feedbacks are found to play important roles in affecting regional air quality in the US by reducing domain-average concentrations of carbon monoxide (CO), O 3 , nitrogen oxide (NO x ), volatile organic compounds (VOCs), and PM 2.5 by 3.1 % (up to 27.8 %), 4.2 % (up to 16.2 %), 6.6 % (up to 50.9 %), 5.8 % (up to 46.6 %), and 8.6 % (up to 49.1 %), respectively, mainly due to reduced radiation, temperature, and wind speed. The overall performance of the two-way coupled WRF-CMAQ model achieved in this work is generally good or satisfactory and the improved performance for two-way coupled WRF-CMAQ should be considered along with other factors in developing future model applications to inform policy making.

58 GEOSCIENCES↗
DOE OSTI2023

Correlated Trajectory Uncertainty for Adaptive Sequential Decision Making

One of the great challenges with decision making tasks on real world systems is the fact that data is sparse and acquiring additional data is expensive. In these cases, it is often crucial to make a model of the environment to assist in making decisions. At the same time, limited data means that learned models are erroneous, making it just as important to equip the model with good predictive uncertainties. In the context of learning sequential decision making policies, these uncertainties can prove useful for informing which data to collect for the greatest improvement in policy performance \citep{mehta2021experimental, mehta2022exploration} or informing the policy about unsure regions of state and action space to avoid during test time \citep{yu2020mopo}. Additionally, assuming that realistic samples of the environment can be drawn, an adaptable policy can be trained that attempts to make optimal decisions for any given possible instance of the environment \citep{ghosh2022offline, chen2021offline}. In this work, we examine the so-called ``probabilistic neural network'' (PNN) model that is ubiquitous in model-based reinforcement learning (MBRL) works. We argue that while PNN models may have good marginal uncertainties, they form a distribution of non-smooth transition functions. Not only are these samples unrealistic and may hamper adaptability, but we also assert that this leads to poor uncertainty estimates when predicting multiple step trajectory estimates. To address this issue, we propose a simple sampling method that can be implemented on top of pre-existing models.We evaluate our sampling technique on a number of environments, including a realistic nuclear fusion task, and find that, not only do smooth transition function samples produce more calibrated uncertainties, but they also lead to better downstream performance for an adaptive policy.

Offline Reinforcement Learning↗
DOE OSTI2020

A Step Towards Machine Learning-based Coherent Generator Grouping for Emergency Control Applications in Modern Power Grid

A dynamic neural network (NN) based multi-class classifier is proposed for improving online prediction of coherent generator groups (CGGs), following the occurrences of various contingencies in the power grid. This is motivated by the increasing availability of the measurements from phasor measurement units (PMUs) and the number of grouping schemes is limited. The proposed method consists of three steps. First, by performing offline simulations, a library of system dynamic responses characterized by post-contingency rotor angles and speeds of individual generators is obtained. To generate sufficient data, up to N-2 contingencies and the uncertain parameters associated with the power grid including type and location of disturbance and fault clearing times are modeled. Secondly, the training data-set is produced by generating labels for individual contingencies using a hierarchical clustering method based on rotor angle and speed data. Lastly, the dynamic NN models are trained for online applications such as emergency controls and controlled islanding. The proposed method is tested on the standard 16-generator 68-bus system to demonstrate its performance. Furthermore, the impact of the sample data lengths on the CGG numbers is evaluated. It is interesting to observe that the time domain stability behaviors can be determined by examining the changes in the CGG numbers.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗