Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Reimagining Energy Efficiency Resource Standards

Energy Efficiency Resource Standards (EERS) are a long-standing policy used to advance state goals and priorities, such as reliability, affordability and decarbonization. They ensure cost-effective energy efficiency is used to lower energy consumption, reduce peak demand, and enable grid flexibility. Adapting EERS through strategic design, and integrating EERS into broader energy plans and policies, can help states optimize their approach to meet modern and evolving power system needs and achieve sustainable energy outcomes. This report examines how states consider energy efficiency in the context of four policy priorities: providing grid benefits, addressing load growth, reducing emissions, and promoting affordability. It provides examples of states that are pursuing these priorities, reviews the role of energy efficiency in them and identifies opportunities to further promote energy efficiency alongside the priorities. Decisionmakers and stakeholders can draw directly from these examples and findings to design an EERS that maximizes co-benefits, captures synergies, and promotes consideration of the all the possible solutions to advance their goals.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Learning Robust Marking Policies for Adaptive Mesh Refinement

Here in this work, we revisit the marking decisions made in the standard adaptive finite element method (AFEM). Experience shows that a naïve marking policy leads to inefficient use of computational resources for adaptive mesh refinement (AMR). Consequently, using AMR in practice often involves ad-hoc or time-consuming offline parameter tuning to set appropriate parameters for the marking subroutine. To address these practical concerns, we recast AMR as a Markov decision process in which refinement parameters can be selected on-the-fly at run time, without the need for pre-tuning by expert users. In this new paradigm, the refinement parameters are also chosen adaptively via a marking policy that can be optimized using methods from reinforcement learning. We use the Poisson equation to demonstrate our techniques on h- and hp-refinement benchmark problems, and our experiments suggest that superior marking policies remain undiscovered for many classical AFEM applications. Furthermore, an unexpected observation from this work is that marking policies trained on one family of PDEs are sometimes robust enough to perform well on problems far outside the training family. For illustration, we show that a simple hp-refinement policy trained on 2D domains with only a single re-entrant corner can be deployed on far more complicated 2D domains, and even 3D domains, without significant performance loss. For reproduction and broader adoption, we accompany this work with an open-source implementation of our methods.

97 MATHEMATICS AND COMPUTING↗

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Recent studies showed that reinforcement learning (RL) is a promising approach for coordination and control of distributed energy resources (DER) under uncertainties. Many existing RL approaches, including Q-learning and approximate dynamic programming, are based on lookup table methods, which become inefficient when the problem size is large and infeasible when continuous states and actions are involved. In addition, when modeling battery energy storage system (BESS), the loss of life is not reasonably considered into the decision-making process. This paper proposes an innovative deep RL method for DER coordination considering BESS degradation. The proposed deep RL is designed based on an adaptive actor-critic architecture and employs an off-policy deterministic policy gradient method for determining the dispatch operation that minimizes the operation cost and BESS life loss. Case studies were performed to validate the proposed method and demonstrate the effects of incorporating degradation models into control design.

Das, Avijit↗

Centralized Stochastic Optimal Control of Complex Systems

In this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain using the long-run expected average cost per unit time criterion, and we show that the control policy yielding the Pareto optimal solution minimizes the average cost criterion online. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion.

complex systems↗

Network-Cognizant Time-Coupled Aggregate Flexibility of Distribution Systems Under Uncertainties

Increasing integration of distributed energy resources (DERs) within distribution feeders provides unprecedented flexibility at the distribution-transmission interconnection. To exploit this flexibility and to use the capacity potential of aggregate DERs, feasible substation power injection trajectories need to be efficiently characterized. This paper provides an ellipsoidal inner approximation of the set of feasible power injection trajectories at the substation such that for any point in the set, there exists a feasible disaggregation strategy of DERs for any load uncertainty realization. The problem is formulated as one of finding the robust maximum volume ellipsoid inside the flexibility region under uncertainty. Though the problem is NP-hard even in the deterministic case, this paper derives novel approximations of the resulting adaptive robust optimization problem based on optimal second-stage policies. The proposed approach yields less conservative flexibility characterization than existing flexibility region approximation formulations. The efficacy of the proposed method is demonstrated on a realistic distribution feeder.

adaptive robust optimization↗

Learning infinite-horizon average-reward restless multi-action bandits via index awareness

We consider the online restless bandits with average-reward and multiple actions, where the state of each arm evolves according to a Markov decision process (MDP), and the reward of pulling an arm depends on both the current state of the corresponding MDP and the action taken. Since finding the optimal control is typically intractable for restless bandits, existing learning algorithms are often computationally expensive or with a regret bound that is exponential in the number of arms and states. In this paper, we advocate \textit{index-aware reinforcement learning} (RL) solutions to design RL algorithms operating on a much smaller dimensional subspace by exploiting the inherent structure in restless bandits. Specifically, we first propose novel index policies to address dimensionality concerns, which are provably optimal. We then leverage the indices to develop two low-complexity index-aware RL algorithms, namely, (i) GM-R2MAB, which has access to a generative model; and (ii) UC-R2MAB, which learns the model using an upper confidence style online exploitation method. We prove that both algorithms achieve a sub-linear regret that is only polynomial in the number of arms and states. A key differentiator between our algorithms and existing ones stems from the fact that our RL algorithms contain a novel exploitation that leverages our proposed provably optimal index policies for decision-makings.

Xiong, Guojun↗

Equilibrium Control Policies for Markov Chains

The average cost criterion has held great intuitive appeal and has attracted considerable attention. It is widely employed when controlling dynamic systems that evolve stochastically over time by means of formulating an optimization problem to achieve long-term goals efficiently. The average cost criterion is especially appealing when the decision-making process is long compared to other timescales involved, and there is no compelling motivation to select short-term optimization. This paper addresses the problem of controlling a Markov chain so as to minimize the average cost per unit time. Our approach treats the problem as a dual constrained optimization problem. We derive conditions guaranteeing that a saddle point exists for the new dual problem and we show that this saddle point is an equilibrium control policy for each state of the Markov chain. For practical situations with constraints consistent to those we study here, our results imply that recognition of such saddle points may be of value in deriving in real time an optimal control policy.

99 GENERAL AND MISCELLANEOUS↗

Economics of the disintegration of the Greenland ice sheet

Concerns about the impact on large-scale earth systems have taken center stage in the scientific and economic analysis of climate change. The present study analyzes the economic impact of a potential disintegration of the Greenland ice sheet (GIS). The study introduces an approach that combines long-run economic growth models, climate models, and reduced-form GIS models. The study demonstrates that social cost–benefit analysis and damage-limiting strategies can be usefully extended to illuminate issues with major long-term consequences, as well as concerns such as potential tipping points, irreversibility, and hysteresis. A key finding is that, under a wide range of assumptions, the risk of GIS disintegration makes a small contribution to the optimal stringency of current policy or to the overall social cost of climate change. It finds that the cost of GIS disintegration adds less than 5% to the social cost of carbon (SCC) under alternative discount rates and estimates of the GIS dynamics.

54 ENVIRONMENTAL SCIENCES↗

Supporting Building Portfolio Investment and Policy Decision Making through an Integrated Building Utility Data Platform

The American Recovery and Reinvestment Act stimulus funding of 2009 for smart grid projects resulted in the tripling of smart meters deployment. In 2012, the Green Button initiative provided utility customers with access to their real-time1 energy usage. The availability of finely granular data provides an enormous potential for energy data analytics and energy benchmarking. The sheer volume of time-series utility data from a large number of buildings also poses challenges in data collection, quality control, and database management for rigorous and meaningful analyses. In this paper, we will describe a building portfolio-level data analytics tool for operational optimization, business investment and policy assessment using 15-minute to monthly intervals utility data. The analytics tool is developed on top of the U.S. Department of Energy’s Standard Energy Efficiency Data (SEED) platform, an open source software application that manages energy performance data of large groups of buildings. To support the significantly large volume of granular interval data, we integrated a parallel time-series database to the existing relational database. The time-series database improves on the current utility data input, focusing on real-time data collection, storage, analytics and data quality control. The fully integrated data platform supports APIs for utility apps development by third party software developers. These apps will provide actionable intelligence for building owners and facilities managers. Unlike a commercial system, this platform is an open source platform funded by the U.S. Government, accessible to the public, researchers and other developers, to support initiatives in reducing building energy consumption.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

CRITICAL FUNCTION ASSURANCE: Understanding Critical Function and Critical Function Delivery is Foundational for Meaningful ICS Security Improvement and Policy Efforts

Modern life is enabled by a complex and interdependent web of critical functions, including energy, communications, transportation, food, and water. Automation has significantly reduced or replaced human interactions in the delivery of these functions, resulting in a web of goods and services that are made available 24/7 only through unique and intentional deployments of microprocessors, software, and firmware technologies. The prospect of cyber-enabled sabotage of these processes disrupts traditional risk determination models. Critical Function Assurance (CFA) is a foundational approach to identifying, prioritizing, and mitigating the risk that is inherent in the delivery of critical functions that depend on digital technology. It provides rapid focus to what matters most and illuminates elements and areas of risk that otherwise are often overlooked. This focus enables effective application of available security resources and optimizes security strategy and policy efforts. This paper introduces CFA to decision makers and risk executives (including CEOs, COOs, CFOs, and CISOs) whose organizations support and deliver the critical functions that underpin national defense, societal health and safety, and a vibrant economy.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Optimal Operations Management of Mobility-on-Demand Systems

The emergence of the sharing economy in urban transportation networks has enabled new fast, convenient and accessible mobility services referred to as Mobilty-on-Demand systems (e.g., Uber, Lyft, DiDi). These platforms have flourished in the last decade around the globe and face many operational challenges in order to be competitive and provide good quality of service. A crucial step in the effective operation of these systems is to reduce customers' waiting time while properly selecting the optimal fleet size and pricing policy. In this paper, we jointly tackle three operational decisions: (i) fleet size, (ii) pricing, and (iii) rebalancing, in order to maximize the platform's profit or its customers' welfare. To accomplish this, we first devise an optimization framework which gives rise to a static policy. Then, we elaborate and propose dynamic policies that are more responsive to perturbations such as unexpected increases in demand. We test this framework in a simulation environment using three case studies and leveraging traffic flow and taxi data from Eastern Massachusetts, New York City, and Chicago. Our results show that solving the problem jointly could increase profits between 1% and up to 50%, depending on the benchmark. Moreover, we observe that the proposed fleet size yield utilization of the vehicles in the fleet is around 75% compared to private vehicle utilization of 5%.

Wollenstein-Betech, Salomón↗

Designing workflows for materials characterization

Experimental science is enabled by the combination of synthesis, imaging, and functional characterization organized into evolving discovery loop. Synthesis of new material is typically followed by a set of characterization steps aiming to provide feedback for optimization or discover fundamental mechanisms. However, the sequence of synthesis and characterization methods and their interpretation, or research workflow, has traditionally been driven by human intuition and is highly domain specific. Here, we explore concepts of scientific workflows that emerge at the interface between theory, characterization, and imaging. In this study, we discuss the criteria by which these workflows can be constructed for special cases of multiresolution structural imaging and functional characterization, as a part of more general material synthesis workflows. Some considerations for theory–experiment workflows are provided. We further pose that the emergence of user facilities and cloud labs disrupts the classical progression from ideation, orchestration, and execution stages of workflow development. To accelerate this transition, we propose the framework for workflow design, including universal hyperlanguages describing laboratory operation, ontological domain matching, reward functions and their integration between domains, and policy development for workflow optimization. These tools will enable knowledge-based workflow optimization; enable lateral instrumental networks, sequential and parallel orchestration of characterization between dissimilar facilities; and empower distributed research.

36 MATERIALS SCIENCE↗

Next-Generation Reverse Logistics Networks of Photovoltaic Recycling: Perspectives and Challenges

With the growing adoption of solar energy as a key component of the global energy transition and its new industrial policy (the Inflation Reduction Act and others), the United States is witnessing a significant increase in solar investments. This surge in installations and domestic and reshored manufacturing of solar photovoltaic (PV) panels brings with it a pressing issue: the proper management of end-of-life (EoL) PV panels. As these panels are decommissioned, either due to reaching the end of their lifespans or due to breakage across the various stages of the forward supply chain, it becomes crucial to establish efficient reverse supply chain logistics systems to address the challenges associated with their disposal, while also unlocking the value of the inherent materials that are of critical value for other forward supply chains. This perspective article examines the challenges regarding EoL PV panels and relevant reverse supply chain and logistics networks, and proposes future research directions based on the gaps observed among academic research, industry, and policy-making challenges. We identify the main bottlenecks and hurdles including, among others, the lack of supportive regulations and absence of structured, optimized recycling infrastructure. To this end, it is proposed that the key to achieving a sustainable reverse supply chain network for solar PV panels lies in relentless end-to-end supply chain cost optimization efforts supported by enabling policies. Moreover, it is proposed that designing systematic decision-making modeling frameworks is vital for examining different possibilities and scenarios for state, region or nation-wide optimization of solar PV reverse supply chain networks. Indicative to this effect, we discuss the development of a Resource-Task-Network (RTN)-based model and demonstrate its application and benefits through a case study. We wrap up with conclusions and future research directions.

circular economy↗

Efficient Sampling Allocation Procedures for Optimal Quantile Selection

We propose a dynamic sampling allocation and selection paradigm for finding the alternative with the optimal quantile in a Bayesian framework. Myopic allocation policies (MAPs), analogous to existing methods in classic ranking and selection for selecting the alternative with the optimal mean, and computationally efficient selection policies are derived for selecting the alternative with the optimal quantile. Under certain conditions, we prove that the proposed MAPs and selection procedures are consistent, which means that the best quantile would be eventually correctly selected as the sample size goes to infinity. Numerical experiments demonstrate that the proposed schemes can significantly improve the performance.

Computer Science↗

A Generation-Storage Coordination Dispatch Strategy for Power System Based on Causal Reinforcement Learning

In the backdrop of global energy transformation, power systems integrating high proportions of renewable energy sources are facing unprecedented challenges in operational stability and dispatch efficiency. To address these challenges, this study introduces a generation-storage coordination real-time dispatch strategy based on Causal Power System Dynamic Reinforcement Learning (CPSDRL). Diverging from traditional reinforcement learning approaches, CPSDRL innovatively incorporates causal inference within the state prediction model - the crux of model-based reinforcement learning - thereby establishing the Power Causal Dynamic Model (PCDM). Assisted by the prior knowledge of power systems, the model significantly enhances prediction accuracy and reliability through a two-stage training process. Utilizing PCDM, this study further applies a direct policy search algorithm to optimize the real-time dispatch strategy. Experimental results indicate that the proposed method improves the stability of generation-storage coordination real-time dispatch and exhibits competitive advantages in sample efficiency and computational speed, compared to traditional model-based and model-free reinforcement learning algorithms. This method is expected to enhance the practicality and adaptability of causal reinforcement learning techniques in power system scheduling and control.

causal reinforcement learning↗

A Modified Maximum Entropy Inverse Reinforcement Learning Approach for Microgrid Energy Scheduling

Increasing popularity of integrating distributed energy resources (DERs) into the power system brings a challenge to optimize the microgrid dispatch policy. The reinforcement learning methods suffer from a long-time problem with the theoretical assumption of the objective/reward function for the microgrid system. Although the traditional inverse reinforcement learning (IRL) approaches can solve this problem to some extent, they encounter a limitation of complex computations for state visitation frequency in the large and continuous state space. To alleviate this limitation, we propose a modified maximum entropy IRL (MMIRL) method to extract the reward function from the expert demonstrations for solving the microgrid energy scheduling problem. The proposed MMIRL algorithm is promising in recovering the reward function and learning the dispatch policy compared to conventional approaches. Case studies are performed in an energy arbitrage problem and a microgrid system with DERs. Results substantiate that the proposed MMIRL approach can learn the dispatch policy with more than 99% efficiency and outperforms other comparative methods.

artificial intelligence, reinforcement learning, m↗

Checkpointing Strategies for Shared High-Performance Computing Platforms

Input/output (I/O) from various sources often contend for scarcely available bandwidth. For example, checkpoint/restart (CR) protocols can help to ensure application progress in failure-prone environments. However, CR I/O alongside an application's normal, requisite I/O can increase I/O contention and might negatively impact performance. In this work, we consider different aspects (system-level scheduling policies and hardware) that optimize the overall performance of concurrently executing CR-based applications that share I/O resources. We provide a theoretical model and derive a set of necessary constraints to minimize the global waste on a given platform. Our results demonstrate that Young/Daly's optimal checkpoint interval, despite providing a sensible metric for a single, undisturbed application, is not sufficient to optimally address resource contention at scale. We show that by combining optimal checkpointing periods with contention-aware system-level I/O scheduling strategies, we can significantly improve overall application performance and maximize the platform throughput. Finally, we evaluate how specialized hardware, namely burst buffers, may help to mitigate the I/O contention problem. Altogether, these results provide critical analysis and direct guidance on how to design efficient, CR ready, large -scale platforms without a large investment in the I/O subsystem.

97 MATHEMATICS AND COMPUTING↗