Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Reinforcement Learning Approach to Flight Control Allocation with Distributed Electric Propulsion

The flight control system of the SUSAN Electrofan concept aircraft achieves attitude control using both conventional flight control surfaces and differential thrust through distributed electric propulsion (DEP) from sixteen wing-mounted electric engines. The introduction of eight pairs of wing fans for attitude control creates a highly actuated system. Such a system requires more sophisticated control to operate, especially in the presence of wingfan failures where the loss of a single wingfan can result in a thrust imbalance. This paper investigates the use of deep reinforcement learning (RL) using proximal policy optimization (PPO) to achieve attitude control through a combination of DEP and control surface deflections. First, the paper examines the aircraft undergoing a coordinated turn. Then, it examines the aircraft experiencing a wingfan failure during cruise conditions. It is shown that deep reinforcement learning can be a potential avenue for nonlinear flight control design.

Distributed Electric Propulsion↗
DOE OSTI2015

Microgrid Sizing Capability v 1.0

The MSC is an optimization model, written in Python, that is used to make an initial determination of the types and quantities of electric power generation and energy storage technologies that should be purchased when developing a microgrid. In addition to determining which technologies to purchase, the model determines the optimal policies for operating the microgrid. The model selects the purchases and operating polices in a manner that minimizes annual energy costs (operating costs plus annualized capital costs), while ensuring that that the purchased technologies are paid back within a reasonable timeframe.

Source record↗
DOE OSTI2022

Cyber Resilient Design and Controls for Energy Systems

As the grid of today is evolving into a highly distributed and autonomous cyber-physical system with higher penetration of Distributed Energy Resources (DERs) and autonomous controls that are dependent on the cyber infrastructure, there are novel and increasing challenges to cyber-resilient operation of the grid. While the traditional mechanisms take a "bolt on" approach to cybersecurity and resiliency, this talk presents two novel technologies developed at NREL that "bake in" security and resilience into the design and control of the grid. An Adaptive Resilience Metrics framework is a novel mechanism for system owners and operators to understand the impact of cyber and physical system design and topology on the resilient operation, while also helping determine optimal policies in response to adverse cyber events. Along with that a zero-knowledge proof-based technique helps incorporate security into the controls layer by focusing on computational integrity rather than data integrity. By leveraging design and control properties unique to Operational Technology systems these technologies will not only help enhance cyber-resilience for the grid of today, but a highly distributed and autonomous grid of tomorrow.

cyber resilience↗
DOE OSTI2026

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

High-throughput scientific facilities such as the Large Hadron Collider depend on real-time event filtering (\textit{triggering}) under tight constraints on bandwidth, latency, and storage. In practice, trigger menus are largely static and hand-tuned and can become suboptimal as detector conditions, pileup, and background composition drift over time. We cast online threshold tuning as a sequential decision-making problem: a reinforcement learning agent ingests streaming summaries of recent rates and signal-sensitive features and updates trigger thresholds to maximize signal efficiency while tracking a target background rate within a tolerance band. We adapt Group-Filtered Policy Optimization (GFPO) to streaming control and introduce two variants (GFPO-F, GFPO-FR) that enforce background rate feasibility during training. On a benchmark that emulates realistic collider operation, we study two representative triggers: a total transverse energy ($H_{T}$) trigger sensitive to pileup variation, and an anomaly-detection (AD) trigger based on reconstruction loss for rare or non-standard signatures. On Monte Carlo streams, our agent increases the fraction of in-tolerance time intervals by 48% ($H_T$) and 28% (AD), with a cumulative gain of up to 2% in signal efficiency on those in-tolerance intervals. Transferring from simulation to \emph{real} collision data (CMS Run 283408), the same agent, without fine-tuning, achieves a 56% ($H_T$) and 28% (AD) in-tolerance improvement over baselines, with further signal-efficiency gain on both triggers. To our knowledge, this is the \emph{first} demonstration of RL-based trigger control on real Large Hadron Collider collision data. Code is available at https://github.com/Zixind/GFPO_LHC (see repo for details).

Ding, Zixin [Chicago U.]↗
DOE OSTI2025

Bridging the Gap: User-Centric Energy Monitoring for Policy-Driven Application Optimization in HPC Data Centers

Application energy optimization in HPC data centers face two critical gaps. Systematic methodologies that connect data center policies to application decisions and accessible monitoring tools that enable data-driven optimization. We address both gaps through two complementary pillars. First, we present a methodology based on extended weighted Energy Delay Product (EDP) to translate data center operational priorities and integrate energy considerations into the energy optimization workflow which starts from continuous monitoring through targeted optimization. Second, we present a user-space monitoring tool, Omnistat, that enables this methodology by providing developers with direct access to actionable energy telemetry. Through deployment on the Frontier supercomputer and case studies exploring performance-energy trade-offs, we show how these pillars help energy as an integral optimization target for developers as active participants in data center efficiency.

Shin, Woong [ORNL] (ORCID:0000000172077814)↗
DOE OSTI2020

Transferable Reinforcement Learning for Smart Homes: Preprint

To harness the great amount of untapped resources at the demand side, smart home technology plays a vital role in solving the "last mile" problem in smart grid. Reinforcement learning (RL), which has demonstrated an outstanding performance in solving many sequential decision-making problems, can be a great candidate to be used in smart home control. For instance, many studies have started investigating the load scheduling problem under dynamic pricing scheme. Based on those, this study aims at providing an affordable solution to encourage a higher smart home adoption rate. Specifically, we investigate combining transfer learning (TL) with RL to reduce the training cost of an optimal RL control policy. Given an optimal policy for a benchmark home, TL can jump-start the RL training of a policy for a new home, which has different appliances and user preferences. Simulation results show that by leveraging TL, RL training converges faster and requires much less computing time for new homes that are similar to the benchmark home. In all, this study proposes a cost-effective approach for training RL control policies for homes at scale, which ultimately reduces the controller's implementation costs, increases the adoption rate of RL controllers, and makes more homes grid-interactive.

ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION,↗
DOE OSTI2022

Combining Simulation and Optimization to Derive Operating Policies for a Concentrating Solar Power Plant

Optimizing short-term decisions over a rolling horizon and/or using deterministic penalties to capture system stochasticity can lead to myopic policies that fail to consider unplanned events and their long-term adverse effects. We present a methodology that integrates an off-line optimization model with a simulation procedure to determine the profitability of different operating strategies; specifically, the latter is used to generate additional constraints for the former when failures occur according to system component operating lifetimes that (i) are subject to exogenous uncertainty, and (ii) may degrade more quickly under specific operating conditions. We use the feedback provided by the simulation model in a parametric analysis to obtain penalties that can be used in short-term operations scheduling to maximize the long-term revenues obtained by the optimization model. We apply this research to a concentrating solar power plant; our results show that the methodology can be used to choose an operating policy that balances maximizing profit while accounting for maintenance costs. Integrating the optimization model with a simulation procedure reveals that aggressive prices for cycling yield about 55% fewer startups and 30% fewer failures compared to using a more typical start-up operating strategy, and can save hundreds of thousands to millions of dollars in repair costs over the lifetime of the plant.

concentrating solar power↗
DOE OSTI2022

Optimal economic dispatch policy for prosumer with energy storage considering self-consumption demand

This paper analyzed the effects of self-consumption demand on the joint economic dispatch of prosumers (energy consumers who are also producers), particularly for prosumers with both energy storage and distributed energy sources (DERs). Studies in the existing literature on the economic dispatch scheduling policy of energy storage, mostly from the perspective of electricity merchants, do not address the impacts of self-consumption demand. However, due to the intermittent and high levels of uncertainty regarding DERs generation and the dynamic demand of the prosumer, production and consumption are not always simultaneous; there are two possible scenarios in each period depending on whether DERs generation can meet prosumers' self-consumption or not. Incorporating the self-consumption demand will pose modeling challenges since these two scenarios cannot occur simultaneously in each period, and different scenarios require different decisions for prosumers. Further, this paper analyzed the two scenarios separately to find the optimal storage scheduling strategy, and the results were combined to get the optimal global solution. We focused on prosumers' economic decision-making while considering self-consumption demand and the physical constraints of a battery based on dynamic programming. Our study showed that the feasible state of charge (SOC) range of storage can be segmented into several sub-ranges by SOC reference points under the above two scenarios. As a result, a prosumer's optimal scheduling can be uniquely and conveniently selected based on the sub-ranges within which the current SOC falls. The results, therefore, provided multistage decision-making guidance for prosumers with energy storage.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Multi-Agent Safe Policy Learning for Power Management of Networked Microgrids

This paper presents a supervised multi-agent safe policy learning (SMAS-PL) method for optimal power management of networked microgrids (MGs) in distribution systems. While unconstrained reinforcement learning (RL) algorithms are black-box decision models that could fail to satisfy grid operational constraints, our proposed method considers AC power flow equations and other operational limits. Accordingly, the training process employs the gradient information of operational constraints to ensure that the optimal control policy functions generate safe and feasible decisions. Furthermore, we have developed a distributed consensus-based optimization approach to train the agents’ policy functions while maintaining MGs’ privacy and data ownership boundaries. After training, the learned optimal policy functions can be safely used by the MGs to dispatch their local resources, without the need to solve a complex optimization problem from scratch. Lastly, numerical experiments have been devised to verify the performance of the proposed method.

, policy gradient↗
DOE OSTI2021

Deep reinforcement learning control of hydraulic fracturing

Hydraulic fracturing is a technique to extract oil and gas from shale formations, and obtaining a uniform proppant concentration along the fracture is key to its productivity. Recently, various model predictive control schemes have been proposed to achieve this objective. But such controllers require an accurate and computationally efficient model which is difficult to obtain given the complexity of the process and uncertainties in the rock formation properties. In this article, we design a model-free data-based reinforcement learning controller which learns an optimal control policy through interactions with the process. Deep reinforcement learning (DRL) controller is based on the Deep Deterministic Policy Gradient algorithm that combines Deep-Q-network with actor-critic framework. In addition, we utilize dimensionality reduction and transfer learning to quicken the learning process. We show that the controller learns an optimal policy to obtain uniform proppant concentration despite the complex nature of the process while satisfying various input constraints.

42 ENGINEERING↗
DOE OSTI2022

An Evolutionary Algorithm to Personalize Stool-Based Colorectal Cancer Screening

Fecal immunochemical testing (FIT) is an established method for colorectal cancer (CRC) screening. Measured FIT-concentrations are associated with both present and future risk of CRC, and may be used for personalized screening. However, evaluation of personalized screening is computationally challenging. In this study, a broadly applicable algorithm is presented to efficiently optimize personalized screening policies that prescribe screening intervals and FIT-cutoffs, based on age and FIT-history. We present a mathematical framework for personalized screening policies and a bi-objective evolutionary algorithm that identifies policies with minimal costs and maximal health benefits. The algorithm is combined with an established microsimulation model (MISCAN-Colon), to accurately estimate the costs and benefits of generated policies, without restrictive Markov assumptions. The performance of the algorithm is demonstrated in three experiments. In Experiment 1, a relatively small benchmark problem, the optimal policies were known. The algorithm approached the maximum feasible benefits with a relative difference of 0.007%. Experiment 2 optimized both intervals and cutoffs, Experiment 3 optimized cutoffs only. Optimal policies in both experiments are unknown. Compared to policies recently evaluated for the USPSTF, personalized screening increased health benefits up to 14 and 4.3%, for Experiments 2 and 3, respectively, without adding costs. Generated policies have several features concordant with current screening recommendations. The method presented in this paper is flexible and capable of optimizing personalized screening policies evaluated with computationally-intensive but established simulation models. It can be used to inform screening policies for CRC or other diseases. For CRC, more debate is needed on what features a policy needs to exhibit to make it suitable for implementation in practice.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

An Edge-Cloud Integrated Solution for Buildings Demand Response Using Reinforcement Learning

Buildings, as major energy consumers, can provide great untapped demand response (DR) resources for grid services. However, their participation remains low in real-life. One major impediment for popularizing DR in buildings is the lack of cost-effective automation systems that can be widely adopted. Existing optimization-based smart building control algorithms suffer from high costs on both building-specific modeling and on-demand computing resources. To tackle these issues, this paper proposes a cost-effective edge-cloud integrated solution using reinforcement learning (RL). Beside RL’s ability to solve sequential optimal decision-making problems, its adaptability to easy-to-obtain building models and the off-line learning feature are likely to reduce the controller’s implementation cost. Using a surrogate building model learned automatically from building operation data, an RL agent learns an optimal control policy on cloud infrastructure, and the policy is then distributed to edge devices for execution. Simulation results demonstrate the control efficacy and the learning efficiency in buildings of different sizes. A preliminary cost analysis on a 4-zone commercial building shows the annual cost for optimal policy training is only 2.25% of the DR incentive received. Results of this study show a possible approach with higher return on investment for buildings to participate in DR programs.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1983

Optimal design and use of retry in fault tolerant real-time computer systems

A new method to determin an optimal retry policy and for use in retry of fault characterization is presented. An optimal retry policy for a given fault characteristic, which determines the maximum allowable retry durations to minimize the total task completion time was derived. The combined fault characterization and retry decision, in which the characteristics of fault are estimated simultaneously with the determination of the optimal retry policy were carried out. Two solution approaches were developed, one based on the point estimation and the other on the Bayes sequential decision. The maximum likelihood estimators are used for the first approach, and the backward induction for testing hypotheses in the second approach. Numerical examples in which all the durations associated with faults have monotone hazard functions, e.g., exponential, Weibull and gamma distributions are presented. These are standard distributions commonly used for modeling analysis and faults.

Lee, Y. H.↗
NASA NTRS1986

Statistical methodologies for the control of dynamic remapping

Following an initial mapping of a problem onto a multiprocessor machine or computer network, system performance often deteriorates with time. In order to maintain high performance, it may be necessary to remap the problem. The decision to remap must take into account measurements of performance deterioration, the cost of remapping, and the estimated benefits achieved by remapping. We examine the tradeoff between the costs and the benefits of remapping two qualitatively different kinds of problems. One problem assumes that performance deteriorates gradually, the other assumes that performance deteriorates suddenly. We consider a variety of policies for governing when to remap. In order to evaluate these policies, statistical models of problem behaviors are developed. Simulation results are presented which compare simple policies with computationally expensive optimal decision policies; these results demonstrate that for each problem type, the proposed simple policies are effective and robust.

Saltz, J. H.↗
DOE OSTI2022

Two-Stage Reinforcement Learning Policy Search for Grid-Interactive Building Control

This paper develops an intelligent grid-interactive building controller, which optimizes building operation during both normal hours and demand response (DR) events. To avoid costly on-demand computation and to adapt to non-linear building models, the controller utilizes reinforcement learning (RL) and makes real-time decisions based on a near-optimal control policy. Learning such a policy typically amounts to solving a hard non-convex optimization problem. We propose to address this problem with a novel global-local policy search method. In the first stage, an RL algorithm based on zero-order gradient estimation is leveraged to search for the optimal policy globally, due to its scalability and the potential to escape some poor performing local optima. The obtained policy is then fine-tuned locally to bring the first-stage solution closer to that of the original unsmoothed problem. Experiments on a simulated five-zone commercial building demonstrate the advantages of the proposed method over existing learning approaches. They also show that the learned control policy outperforms a pragmatic linear model predictive controller (MPC) and approaches the performance of an oracle MPC in testing scenarios. Using a state-of-the-art advanced computing system, we demonstrate that the controller can be learned and deployed within hours of training.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Improved microgrid resiliency through distributionally robust optimization under a policy-mode framework

Critical energy infrastructure are constantly under stress due to the ever increasing disruptions caused by wildfires, hurricanes, other weather related extreme events and cyber-attacks. Hence it becomes important to make critical infrastructure resilient to threats from such cyber-physical events. However, such events are hard to predict and numerous in nature and type and it becomes infeasible to make a system resilient to every possible such cyber-physical event. Such an approach can make the system operation overly conservative and impractical to operate. Furthermore, distributions of such events are hard to predict and historical data available on such events can be very sparse, making the problem even harder to solve. To deal with these issues, in this paper we present a policy-mode framework that enumerates and predicts the probability of various cyber-physical events and then a distributionally robust optimization (DRO) formulation that is robust to the sparsity of the available historical data. The proposed algorithm is illustrated on an islanded microgrid example: a modified IEEE 123-node feeder with distributed energy resources (DERs) and energy storage. Simulations are carried to validate the resiliency metrics under the sampled disruption events.

Nazir, Mohammad Nawaf↗
NASA NTRS1972

On the determination of optimal costly measurement strategies for linear stochastic systems.

This paper presents the formulation of a class of optimization problems dealing with selecting, at each instant of time, one measurement provided by one out of many sensors. Each measurement has an associated measurement cost. The basic problem is then to select an optimal measurement policy, during a specified observation time interval, so that a weighted combination of prediction accuracy and accumulated observation cost is optimized. The current analysis is limited to the class of linear stochastic dynamic systems and measurement subsystems. The problem of selecting the optimal measurement strategy can be transformed into a deterministic optimal control problem. It is shown that the optimal measurement policy and the associated matched Kalman-type filter can be precomputed.

Athans, M.↗