Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A Generation-Storage Coordination Dispatch Strategy for Power System Based on Causal Reinforcement Learning

In the backdrop of global energy transformation, power systems integrating high proportions of renewable energy sources are facing unprecedented challenges in operational stability and dispatch efficiency. To address these challenges, this study introduces a generation-storage coordination real-time dispatch strategy based on Causal Power System Dynamic Reinforcement Learning (CPSDRL). Diverging from traditional reinforcement learning approaches, CPSDRL innovatively incorporates causal inference within the state prediction model - the crux of model-based reinforcement learning - thereby establishing the Power Causal Dynamic Model (PCDM). Assisted by the prior knowledge of power systems, the model significantly enhances prediction accuracy and reliability through a two-stage training process. Utilizing PCDM, this study further applies a direct policy search algorithm to optimize the real-time dispatch strategy. Experimental results indicate that the proposed method improves the stability of generation-storage coordination real-time dispatch and exhibits competitive advantages in sample efficiency and computational speed, compared to traditional model-based and model-free reinforcement learning algorithms. This method is expected to enhance the practicality and adaptability of causal reinforcement learning techniques in power system scheduling and control.

causal reinforcement learning↗
DOE OSTI2023

A Modified Maximum Entropy Inverse Reinforcement Learning Approach for Microgrid Energy Scheduling

Increasing popularity of integrating distributed energy resources (DERs) into the power system brings a challenge to optimize the microgrid dispatch policy. The reinforcement learning methods suffer from a long-time problem with the theoretical assumption of the objective/reward function for the microgrid system. Although the traditional inverse reinforcement learning (IRL) approaches can solve this problem to some extent, they encounter a limitation of complex computations for state visitation frequency in the large and continuous state space. To alleviate this limitation, we propose a modified maximum entropy IRL (MMIRL) method to extract the reward function from the expert demonstrations for solving the microgrid energy scheduling problem. The proposed MMIRL algorithm is promising in recovering the reward function and learning the dispatch policy compared to conventional approaches. Case studies are performed in an energy arbitrage problem and a microgrid system with DERs. Results substantiate that the proposed MMIRL approach can learn the dispatch policy with more than 99% efficiency and outperforms other comparative methods.

artificial intelligence, reinforcement learning, m↗
DOE OSTI2019

Checkpointing Strategies for Shared High-Performance Computing Platforms

Input/output (I/O) from various sources often contend for scarcely available bandwidth. For example, checkpoint/restart (CR) protocols can help to ensure application progress in failure-prone environments. However, CR I/O alongside an application's normal, requisite I/O can increase I/O contention and might negatively impact performance. In this work, we consider different aspects (system-level scheduling policies and hardware) that optimize the overall performance of concurrently executing CR-based applications that share I/O resources. We provide a theoretical model and derive a set of necessary constraints to minimize the global waste on a given platform. Our results demonstrate that Young/Daly's optimal checkpoint interval, despite providing a sensible metric for a single, undisturbed application, is not sufficient to optimally address resource contention at scale. We show that by combining optimal checkpointing periods with contention-aware system-level I/O scheduling strategies, we can significantly improve overall application performance and maximize the platform throughput. Finally, we evaluate how specialized hardware, namely burst buffers, may help to mitigate the I/O contention problem. Altogether, these results provide critical analysis and direct guidance on how to design efficient, CR ready, large -scale platforms without a large investment in the I/O subsystem.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Network-Cognizant Time-Coupled Aggregate Flexibility of Distribution Systems Under Uncertainties

Increasing integration of distributed energy resources (DERs) within distribution feeders provides unprecedented flexibility at the distribution-transmission interconnection. To exploit this flexibility and to use the capacity potential of aggregate DERs, feasible substation power injection trajectories need to be efficiently characterized. This paper provides an ellipsoidal inner approximation of the set of feasible power injection trajectories at the substation such that for any point in the set, there exists a feasible disaggregation strategy of DERs for any load uncertainty realization. The problem is formulated as one of finding the robust maximum volume ellipsoid inside the flexibility region under uncertainty. Though the problem is NP-hard even in the deterministic case, this paper derives novel approximations of the resulting adaptive robust optimization problem based on optimal second-stage policies. The proposed approach yields less conservative flexibility characterization than existing flexibility region approximation formulations. The efficacy of the proposed method is demonstrated on a realistic distribution feeder.

aggregates↗
NASA NTRS2008

Problems With Deployment of Multi-Domained, Multi-Homed Mobile Networks

This document describes numerous problems associated with deployment of multi-homed mobile platforms consisting of multiple networks and traversing large geographical areas. The purpose of this document is to provide insight to real-world deployment issues and provide information to groups that are addressing many issues related to multi-homing, policy-base routing, route optimization and mobile security - particularly those groups within the Internet Engineering Task Force.

Ivancic, William D.↗
DOE OSTI2021

Implication of production tax credit on economic dispatch for electricity merchants with storage and wind farms

The production tax credit (PTC) promotes wind energy development, reduces power generation costs, and can affect merchants’ joint economic dispatch, particularly for electricity merchants with both energy storage and wind farms. Two common PTC policies are studied – in the first policy, a wind farm receives PTC by selling wind generation to the market and its storage can be used to store energy from the wind generation and energy purchased from the grid but the energy released from the storage cannot receive PTC; in the second policy, the energy released from the storage can also qualify for PTC but purchasing energy from the grid is not allowed. We then employ dynamic programming to study merchants' optimal decision-making while considering PTC and the physical characteristics of storage systems. We analytically show that the state of charge (SOC) range can be segmented into different regions by SOC reference points under two PTC policies. The merchant's optimal action can be conveniently and uniquely determined based on the region within which the current SOC falls. Moreover, this study illustrates that PTC could substantially alter the optimal scheduling policy structures by affecting reference points and their relationships. The results showed that the frequencies for charging and discharging storage decisions decreased with an increase in PTC subsidy. Last, we confirm that, although the first policy allows merchants to buy electricity from the market, the second policy can bring more profits when the PTC is large at the current PTC rates. Furthermore, the findings can provide multistage decision-making guidance to electricity merchants in the wholesale power market.

17 WIND ENERGY↗
DOE OSTI2015

Energy Efficiency Under Alternative Carbon Policies. Incentives, Measurement, and Interregional Effects

In this report, we examine and compare how tradable mass-based polices and tradable rate-based policies create different incentives for energy efficiency investments. Through a generalized demonstration and set of examples, we show that as a result of the output subsidy they create, traditional rate-based policies, those that do not credit energy savings from efficiency measures, reduce the incentive for investment in energy efficiency measures relative to an optimally designed mass-based policy or equivalent carbon tax. We then show that this reduced incentive can be partially addressed by modifying the rate-based policy such that electricity savings from energy efficiency measures are treated as a source of zero-carbon generation within the framework of the standard, or equivalently, by assigning avoided emissions credit to the electricity savings at the rate of the intensity target. These approaches result in an extension of the output subsidy to efficiency measures and eliminate the distortion between supply-side and demand-side options for GHG emissions reduction. However, these approaches do not address electricity price distortions resulting from the output subsidy that also impact the value of efficiency measures. Next, we assess alternative approaches for crediting energy efficiency savings within the framework of a rate-based policy. Finally, we identify a number of challenges that arise in implementing a rate-based policy with efficiency crediting, including the requirement to develop robust estimates of electricity savings in order to assess compliance, and the requirement to track the regionality of the generation impacts of efficiency measures to account for their interstate effects.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

A tri-level optimization model for interdependent infrastructure network resilience against compound hazard events

Resilient operation of interdependent infrastructures against compound hazard events is essential for maintaining societal well-being. To address consequence assessment challenges in this problem space, we propose a novel policy-guided tri-level optimization model applied to a proof-of-concept case study with fuel distribution and transportation networks – encompassing one realistic network; one fictitious, yet realistic network; as well as networks drawn from three synthetic distributions. Mathematically, our approach takes the form of a defender-attacker-defender (DAD) model—a multi-agent tri-level optimization, comprised of a defender, attacker, and an operator acting in sequence. Here, in this study, our notional operator may choose proxy actions to operate an interdependent system comprised of fuel terminals and gas stations (functioning as supplies) and a transportation network with traffic flow (functioning as demand) to minimize unmet demand at gas stations. A notional attacker aims to hypothetically disrupt normal operations by reducing supply at the supply terminals, and the notional defender aims to identify best proxy defense policy options which include hardening supply terminals or allowing alternative distribution methods such as trucking reserve supplies. We solve our DAD formulation at a metropolitan scale and present practical defense policy insights against hypothetical compound hazards. We demonstrate the generalizability of our framework by presenting results for a realistic network; a fictitious, yet realistic network; as well as for three networks drawn from synthetic distributions. Additionally, we demonstrate the scalability of the framework by investigating runtime performance as a function of the network size. Steps for future research are also discussed.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

We present differentiable predictive control (DPC), a method for offline learning of constrained neural control policies for nonlinear dynamical systems with performance guarantees. We show that the sensitivities of the parametric optimal control problem can be used to obtain direct policy gradients. Specifically, we employ automatic differentiation (AD) to efficiently compute the sensitivities of the model predictive control (MPC) objective function and constraints penalties. To guarantee safety upon deployment, we derive probabilistic guarantees on closed-loop stability and constraint satisfaction based on indicator functions and Hoeffding’s inequality. We empirically demonstrate that the proposed method can learn neural control policies for various parametric optimal control tasks. In particular, we show that the proposed DPC method can stabilize systems with unstable dynamics, track time-varying references, and satisfy nonlinear state and input constraints. Our DPC method has practical time savings compared to alternative approaches for fast and memory-efficient controller design. Specifically, DPC does not depend on a supervisory controller as opposed to approximate MPC based on imitation learning. We demonstrate that, without losing performance, DPC is scalable with greatly reduced demands on memory and computation compared to implicit and explicit MPC while being more sample efficient than model-free reinforcement learning (RL) algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Real-Time Ecodriving Control in Electrified Connected and Autonomous Vehicles Using Approximate Dynamic Programing

Connected and automated vehicles (CAVs), particularly those with a hybrid electric powertrain, have the potential to significantly improve vehicle energy savings in real-world driving conditions. In particular, the ecodriving problem seeks to design optimal speed and power usage profiles based on available information from connectivity and advanced mapping features to minimize the fuel consumption over an itinerary. This paper presents a hierarchical multilayer model predictive control (MPC) approach for improving the fuel economy of a 48 V mild-hybrid powertrain in a connected vehicle environment. Approximate dynamic programing (DP) is used to solve the receding horizon optimal control problem, whose terminal cost is approximated with the base policy obtained from the long-term optimization. The controller was tested virtually (with deterministic and Monte Carlo simulation) across multiple real-world routes, demonstrating energy savings of more than 20%. The controller was then deployed on a test vehicle equipped with a rapid prototyping embedded controller. In-vehicle testing confirm the energy savings obtained in simulation and demonstrate the real-time ability of the controller.

Automation & Control Systems↗
DOE OSTI2015

A model of interaction between anticorruption authority and corruption groups

The paper provides a model of interaction between anticorruption unit and corruption groups. The main policy functions of the anticorruption unit involve reducing corrupt practices in some entities through an optimal approach to resource allocation and effective anticorruption policy. We develop a model based on Markov decision-making process and use Howard’s policy-improvement algorithm for solving an optimal decision strategy. We examine the assumption that corruption groups retaliate against the anticorruption authority to protect themselves. This model was implemented through stochastic game.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Reliability-informed end-of-use decision making for product sustainability using two-stage stochastic optimization

The concept of circular economy has been diffused in recent decades to promote economic growth that does not add to the burden on natural resource extraction. Re-X options (e.g., reuse, repair, refurbish, remanufacture, recycle) have been gradually adopted in the product development process and optimized to reduce or eliminate waste and pollution. Although manufacturing incorporating Re-X options can be more environmentally friendly, it involves more sources of uncertainty than traditional manufacturing since the end-of-use products can be collected from multiple origins with various quantities and qualities, and the market demand for both new and remanufactured products cannot be forecasted perfectly. Thus, there is a need to optimize the Re-X policy to alleviate the negative impacts of the higher uncertainty. One option is using the reliability information of new products to estimate the end-of-use conditions and applying multi-stage stochastic optimization to capture multiple demand scenarios. This paper develops a two-stage stochastic optimization model to optimize the quality thresholds for reuse, recycling, and remanufacturing options. Our objective is to minimize the total cost, energy consumption, and environmental impact of producing and providing warranty service for a product family. The model employs reliability information of product components to estimate the warranty service cost and the end-of-use conditions of the returned resources. A case study on a general product family is implemented to illustrate the efficacy of the optimization model. Finally, results show that the two-stage optimization can achieve cost and environmental impact reduction for a hybrid manufacturing and remanufacturing process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Dataset For: A Guide to Residential Energy Storage and Rooftop Solar: State Net Metering Policies and Utility Rate Tariff Structures

Federal and state decarbonization goals have led to numerous financial incentives and policies designed to increase access and adoption of renewable energy systems. In combination with the declining cost of both solar photovoltaic and battery energy storage systems and rising electric utility rates, residential renewable adoption has become more favorable than ever. However, not all states provide the same opportunity for cost recovery, and the complicated and changing policy and utility landscape can make it difficult for households to make an informed decision on whether to install a renewable system. This paper is intended to provide a guide to households considering renewable adoption by introducing relevant factors that influence renewable system performance and payback, summarized in a state lookup table for quick reference. Five states are chosen as case studies to perform economic optimizations based on net metering policy, utility rate structure, and average electric utility price; these states are selected to be representative of the possible combinations of factors to aid in the decision-making process for customers in all states. The results of this analysis highlight the dual importance of both state support for renewables and price signals, as the benefits of residential renewable systems are best realized in states with net metering policies facing the challenge of above-average electric utility rates. This dataset is intended to allow readers to reproduce and customize the analysis performed in this work to their benefit. Suggested modifications include: location, household load profile, rate tariff structure, and renewable energy system design.

14 SOLAR ENERGY↗
DOE OSTI2023

To what extent can decommissioning options for marine artificial structures move us toward environmental targets?

Switching from fossil fuels to renewable energy is key to international energy transition efforts and the move toward net zero. For many nations, this requires decommissioning of hundreds of oil and gas infrastructure in the marine environment. Current international, regional and national legislation largely dictates that structures must be completely removed at end-of-life although, increasingly, alternative decommissioning options are being promoted and implemented. Yet, a paucity of real-world case studies describing the impacts of decommissioning on the environment make decision-making with respect to which option(s) might be optimal for meeting international and regional strategic environmental targets challenging. To address this gap, we draw together international expertise and judgment from marine environmental scientists on marine artificial structures as an alternative source of evidence that explores how different decommissioning options might ameliorate pressures that drive environmental status toward (or away) from environmental objectives. Synthesis reveals that for 37 United Nations and Oslo-Paris Commissions (OSPAR) global and regional environmental targets, experts consider repurposing or abandoning individual structures, or abandoning multiple structures across a region, as the options that would most strongly contribute toward targets. This collective view suggests complete removal may not be best for the environment or society. However, different decommissioning options act in different ways and make variable contributions toward environmental targets, such that policy makers and managers would likely need to prioritise some targets over others considering political, social, economic, and ecological contexts. Current policy may not result in optimal outcomes for the environment or society.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1981

Spacecraft computer resource margin management

The conduction of the Project Galileo Orbiter, with 18 microcomputers and the equivalent of 360K 8-bit bytes of memory contained within two major engineering subsystems and eight science instruments, requires that the key onboard computer system resources be managed in a very rigorous manner. Attention is given to the rationale behind the project policy, the development stage, the preliminary design stage, the design/implementation stage, and the optimization or 'scrubbing' stage. The implementation of the policy is discussed, taking into account the development of the Attitude and Articulation Control Subsystem (AACS) and the Command and Data Subsystem (CDS), the reporting of margin status, and the response to allocation oversubscription.

Larman, B. T.↗
DOE OSTI2026

A transfer learning approach to energy-efficient control of small and medium-sized commercial buildings

Model-free reinforcement learning (RL) provides a data-driven and adaptive approach to optimize building energy use while satisfying occupant comfort. This powerful tool does not need any prior knowledge about the environment and system it is optimizing and can adapt its policy based on the changes in captures. Like any other data-driven tool, it faces high training costs due to the extensive agent-environment interactions required to capture long-term building dynamics and user comfort. Transfer learning, particularly policy distillation, offers a promising way to accelerate training by leveraging pretrained RL agents in different building and system types. Here, this study investigates online student distillation, in which the student model updates its neural network weights using outputs from teacher models. The work introduces a student distillation strategy designed for efficient knowledge transfer, along with a teacher selection method that ensures high-quality guidance. The approach is validated using a highly calibrated whole building energy model for a small/medium commercial building test facility. Results show substantial reductions in training time and data requirements while surpassing the performance of ASHRAE Guideline 36, an advanced rule-based control strategy. The distilled RL model required 45% less data and achieved 20% higher cumulative rewards than a state-of-the-art RL model, with faster convergence and lower energy consumption. These outcomes demonstrate that effective transfer learning enables a scalable and data-efficient energy management solution for commercial buildings.

ASHRAE guideline 36↗