Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Multi-Objective Adaptive Mesh Refinement Using Reinforcement Learning

Finite element methods approximate the solution to a partial differential equation (PDE) on a mesh consisting of many elements. In general, using more, smaller elements results in a lower error in the approximation. However, it is often possible to lower the error substantially by only refining, or decreasing the size of, the few elements in the mesh that have the highest error. Adaptive mesh refinement (AMR) is a process that selectively refines regions of a mesh with high error to achieve a desired accuracy in as few degrees of freedom (DOFs) as possible. AMR is favorable compared to uniform refinement, which refines all elements of the mesh equally, because it can often achieve the same accuracy without wasting extra computation time on refinement of elements that already have low error. However, it is difficult to know which elements to refine. In this report we explore ways to choose which elements to refine such that we minimize both the resulting error and the cumulative DOFs used in computation. In particular, we introduce a Pareto-front learning algorithm that trains a policy to give the optimal refinement actions to minimize the cumulative DOFs used to achieve a given target error. Such a policy is useful because it can be deployed on many different problem types where different accuracy levels are desired. Furthermore, training a single policy for a range of target errors allows us to use transfer learning to reduce the required training time.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

A Reinforcement Learning Approach to Parameter Selection for Distributed Optimal Power Flow

With the increasing penetration of distributed energy resources, distributed optimization algorithms have attracted significant attention for power systems applications due to their potential for superior scalability, privacy, and robustness to a single point-of-failure. The Alternating Direction Method of Multipliers (ADMM) is a popular distributed optimization algorithm; however, its convergence performance is highly dependent on the selection of penalty parameters, which are usually chosen heuristically. In this work, we use reinforcement learning (RL) to develop an adaptive penalty parameter selection policy for alternating current optimal power flow (ACOPF) problem solved via ADMM with the goal of minimizing the number of iterations until convergence. We train our RL policy using deep Q-learning and show that this policy can result in significantly accelerated convergence (up to a 59% reduction in the number of iterations compared to existing, curvatureinformed penalty parameter selection methods). Furthermore, we show that our RL policy demonstrates promise for generalizability, performing well under unseen loading schemes as well as under unseen losses of lines and generators (up to a 50% reduction in iterations). This work thus provides a proof-of-concept for using RL for parameter selection in ADMM for power systems applications.

alternating current optimal power flow↗
NASA NTRS1983

Control of a heterogeneous two-server exponential queueing system

A dynamic control policy known as 'threshold queueing' is defined for scheduling customers from a Poisson source on a set of two exponential servers with dissimilar service rates. The slower server is invoked in response to instantaneous system loading as measured by the length of the queue of waiting customers. In a threshold queueing policy, a specific queue length is identified as a 'threshold,' beyond which the slower server is invoked. The slower server remains busy until it completes service on a customer and the queue length is less than its invocation threshold. Markov chain analysis is employed to analyze the performance of the threshold queueing policy and to develop optimality criteria. It is shown that probabilistic control is suboptimal to minimize the mean number of customers in the system. An approximation to the optimum policy is analyzed which is computationally simple and suffices for most operational applications.

Larsen, R. L.↗
DOE OSTI2025

Reimagining Energy Efficiency Resource Standards

Energy Efficiency Resource Standards (EERS) are a long-standing policy used to advance state goals and priorities, such as reliability, affordability and decarbonization. They ensure cost-effective energy efficiency is used to lower energy consumption, reduce peak demand, and enable grid flexibility. Adapting EERS through strategic design, and integrating EERS into broader energy plans and policies, can help states optimize their approach to meet modern and evolving power system needs and achieve sustainable energy outcomes. This report examines how states consider energy efficiency in the context of four policy priorities: providing grid benefits, addressing load growth, reducing emissions, and promoting affordability. It provides examples of states that are pursuing these priorities, reviews the role of energy efficiency in them and identifies opportunities to further promote energy efficiency alongside the priorities. Decisionmakers and stakeholders can draw directly from these examples and findings to design an EERS that maximizes co-benefits, captures synergies, and promotes consideration of the all the possible solutions to advance their goals.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Learning Robust Marking Policies for Adaptive Mesh Refinement

Here in this work, we revisit the marking decisions made in the standard adaptive finite element method (AFEM). Experience shows that a naïve marking policy leads to inefficient use of computational resources for adaptive mesh refinement (AMR). Consequently, using AMR in practice often involves ad-hoc or time-consuming offline parameter tuning to set appropriate parameters for the marking subroutine. To address these practical concerns, we recast AMR as a Markov decision process in which refinement parameters can be selected on-the-fly at run time, without the need for pre-tuning by expert users. In this new paradigm, the refinement parameters are also chosen adaptively via a marking policy that can be optimized using methods from reinforcement learning. We use the Poisson equation to demonstrate our techniques on h- and hp-refinement benchmark problems, and our experiments suggest that superior marking policies remain undiscovered for many classical AFEM applications. Furthermore, an unexpected observation from this work is that marking policies trained on one family of PDEs are sometimes robust enough to perform well on problems far outside the training family. For illustration, we show that a simple hp-refinement policy trained on 2D domains with only a single re-entrant corner can be deployed on far more complicated 2D domains, and even 3D domains, without significant performance loss. For reproduction and broader adoption, we accompany this work with an open-source implementation of our methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Recent studies showed that reinforcement learning (RL) is a promising approach for coordination and control of distributed energy resources (DER) under uncertainties. Many existing RL approaches, including Q-learning and approximate dynamic programming, are based on lookup table methods, which become inefficient when the problem size is large and infeasible when continuous states and actions are involved. In addition, when modeling battery energy storage system (BESS), the loss of life is not reasonably considered into the decision-making process. This paper proposes an innovative deep RL method for DER coordination considering BESS degradation. The proposed deep RL is designed based on an adaptive actor-critic architecture and employs an off-policy deterministic policy gradient method for determining the dispatch operation that minimizes the operation cost and BESS life loss. Case studies were performed to validate the proposed method and demonstrate the effects of incorporating degradation models into control design.

Das, Avijit↗
DOE OSTI2015

Centralized Stochastic Optimal Control of Complex Systems

In this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain using the long-run expected average cost per unit time criterion, and we show that the control policy yielding the Pareto optimal solution minimizes the average cost criterion online. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion.

complex systems↗
DOE OSTI2020

Network-Cognizant Time-Coupled Aggregate Flexibility of Distribution Systems Under Uncertainties

Increasing integration of distributed energy resources (DERs) within distribution feeders provides unprecedented flexibility at the distribution-transmission interconnection. To exploit this flexibility and to use the capacity potential of aggregate DERs, feasible substation power injection trajectories need to be efficiently characterized. This paper provides an ellipsoidal inner approximation of the set of feasible power injection trajectories at the substation such that for any point in the set, there exists a feasible disaggregation strategy of DERs for any load uncertainty realization. The problem is formulated as one of finding the robust maximum volume ellipsoid inside the flexibility region under uncertainty. Though the problem is NP-hard even in the deterministic case, this paper derives novel approximations of the resulting adaptive robust optimization problem based on optimal second-stage policies. The proposed approach yields less conservative flexibility characterization than existing flexibility region approximation formulations. The efficacy of the proposed method is demonstrated on a realistic distribution feeder.

adaptive robust optimization↗
DOE OSTI2022

Learning infinite-horizon average-reward restless multi-action bandits via index awareness

We consider the online restless bandits with average-reward and multiple actions, where the state of each arm evolves according to a Markov decision process (MDP), and the reward of pulling an arm depends on both the current state of the corresponding MDP and the action taken. Since finding the optimal control is typically intractable for restless bandits, existing learning algorithms are often computationally expensive or with a regret bound that is exponential in the number of arms and states. In this paper, we advocate \textit{index-aware reinforcement learning} (RL) solutions to design RL algorithms operating on a much smaller dimensional subspace by exploiting the inherent structure in restless bandits. Specifically, we first propose novel index policies to address dimensionality concerns, which are provably optimal. We then leverage the indices to develop two low-complexity index-aware RL algorithms, namely, (i) GM-R2MAB, which has access to a generative model; and (ii) UC-R2MAB, which learns the model using an upper confidence style online exploitation method. We prove that both algorithms achieve a sub-linear regret that is only polynomial in the number of arms and states. A key differentiator between our algorithms and existing ones stems from the fact that our RL algorithms contain a novel exploitation that leverages our proposed provably optimal index policies for decision-makings.

Xiong, Guojun↗
DOE OSTI2019

Economics of the disintegration of the Greenland ice sheet

Concerns about the impact on large-scale earth systems have taken center stage in the scientific and economic analysis of climate change. The present study analyzes the economic impact of a potential disintegration of the Greenland ice sheet (GIS). The study introduces an approach that combines long-run economic growth models, climate models, and reduced-form GIS models. The study demonstrates that social cost–benefit analysis and damage-limiting strategies can be usefully extended to illuminate issues with major long-term consequences, as well as concerns such as potential tipping points, irreversibility, and hysteresis. A key finding is that, under a wide range of assumptions, the risk of GIS disintegration makes a small contribution to the optimal stringency of current policy or to the overall social cost of climate change. It finds that the cost of GIS disintegration adds less than 5% to the social cost of carbon (SCC) under alternative discount rates and estimates of the GIS dynamics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Supporting Building Portfolio Investment and Policy Decision Making through an Integrated Building Utility Data Platform

The American Recovery and Reinvestment Act stimulus funding of 2009 for smart grid projects resulted in the tripling of smart meters deployment. In 2012, the Green Button initiative provided utility customers with access to their real-time1 energy usage. The availability of finely granular data provides an enormous potential for energy data analytics and energy benchmarking. The sheer volume of time-series utility data from a large number of buildings also poses challenges in data collection, quality control, and database management for rigorous and meaningful analyses. In this paper, we will describe a building portfolio-level data analytics tool for operational optimization, business investment and policy assessment using 15-minute to monthly intervals utility data. The analytics tool is developed on top of the U.S. Department of Energy’s Standard Energy Efficiency Data (SEED) platform, an open source software application that manages energy performance data of large groups of buildings. To support the significantly large volume of granular interval data, we integrated a parallel time-series database to the existing relational database. The time-series database improves on the current utility data input, focusing on real-time data collection, storage, analytics and data quality control. The fully integrated data platform supports APIs for utility apps development by third party software developers. These apps will provide actionable intelligence for building owners and facilities managers. Unlike a commercial system, this platform is an open source platform funded by the U.S. Government, accessible to the public, researchers and other developers, to support initiatives in reducing building energy consumption.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

CRITICAL FUNCTION ASSURANCE: Understanding Critical Function and Critical Function Delivery is Foundational for Meaningful ICS Security Improvement and Policy Efforts

Modern life is enabled by a complex and interdependent web of critical functions, including energy, communications, transportation, food, and water. Automation has significantly reduced or replaced human interactions in the delivery of these functions, resulting in a web of goods and services that are made available 24/7 only through unique and intentional deployments of microprocessors, software, and firmware technologies. The prospect of cyber-enabled sabotage of these processes disrupts traditional risk determination models. Critical Function Assurance (CFA) is a foundational approach to identifying, prioritizing, and mitigating the risk that is inherent in the delivery of critical functions that depend on digital technology. It provides rapid focus to what matters most and illuminates elements and areas of risk that otherwise are often overlooked. This focus enables effective application of available security resources and optimizes security strategy and policy efforts. This paper introduces CFA to decision makers and risk executives (including CEOs, COOs, CFOs, and CISOs) whose organizations support and deliver the critical functions that underpin national defense, societal health and safety, and a vibrant economy.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2021

Optimal Operations Management of Mobility-on-Demand Systems

The emergence of the sharing economy in urban transportation networks has enabled new fast, convenient and accessible mobility services referred to as Mobilty-on-Demand systems (e.g., Uber, Lyft, DiDi). These platforms have flourished in the last decade around the globe and face many operational challenges in order to be competitive and provide good quality of service. A crucial step in the effective operation of these systems is to reduce customers' waiting time while properly selecting the optimal fleet size and pricing policy. In this paper, we jointly tackle three operational decisions: (i) fleet size, (ii) pricing, and (iii) rebalancing, in order to maximize the platform's profit or its customers' welfare. To accomplish this, we first devise an optimization framework which gives rise to a static policy. Then, we elaborate and propose dynamic policies that are more responsive to perturbations such as unexpected increases in demand. We test this framework in a simulation environment using three case studies and leveraging traffic flow and taxi data from Eastern Massachusetts, New York City, and Chicago. Our results show that solving the problem jointly could increase profits between 1% and up to 50%, depending on the benchmark. Moreover, we observe that the proposed fleet size yield utilization of the vehicles in the fleet is around 75% compared to private vehicle utilization of 5%.

Wollenstein-Betech, Salomón↗
DOE OSTI2024

Designing workflows for materials characterization

Experimental science is enabled by the combination of synthesis, imaging, and functional characterization organized into evolving discovery loop. Synthesis of new material is typically followed by a set of characterization steps aiming to provide feedback for optimization or discover fundamental mechanisms. However, the sequence of synthesis and characterization methods and their interpretation, or research workflow, has traditionally been driven by human intuition and is highly domain specific. Here, we explore concepts of scientific workflows that emerge at the interface between theory, characterization, and imaging. In this study, we discuss the criteria by which these workflows can be constructed for special cases of multiresolution structural imaging and functional characterization, as a part of more general material synthesis workflows. Some considerations for theory–experiment workflows are provided. We further pose that the emergence of user facilities and cloud labs disrupts the classical progression from ideation, orchestration, and execution stages of workflow development. To accelerate this transition, we propose the framework for workflow design, including universal hyperlanguages describing laboratory operation, ontological domain matching, reward functions and their integration between domains, and policy development for workflow optimization. These tools will enable knowledge-based workflow optimization; enable lateral instrumental networks, sequential and parallel orchestration of characterization between dissimilar facilities; and empower distributed research.

36 MATERIALS SCIENCE↗
NASA NTRS2003

Route Monopolie and Optimal Nonlinear Pricing

To cope with air traffic growth and congested airports, two solutions are apparent on the supply side: 1) use larger aircraft in the hub and spoke system; or 2) develop new routes through secondary airports. An enlarged route system through secondary airports may increase the proportion of route monopolies in the air transport market.The monopoly optimal non linear pricing policy is well known in the case of one dimension (one instrument, one characteristic) but not in the case of several dimensions. This paper explores the robustness of the one dimensional screening model with respect to increasing the number of instruments and the number of characteristics. The objective of this paper is then to link and fill the gap in both literatures. One of the merits of the screening model has been to show that a great varieD" of economic questions (non linear pricing, product line choice, auction design, income taxation, regulation...) could be handled within the same framework.VCe study a case of non linear pricing (2 instruments (2 routes on which the airline pro_ddes customers with services), 2 characteristics (demand of services on these routes) and two values per characteristic (low and high demand of services on these routes)) and we show that none of the conclusions of the one dimensional analysis remain valid. In particular, upward incentive compatibility constraint may be binding at the optimum. As a consequence, they may be distortion at the top of the distribution. In addition to this, we show that the optimal solution often requires a kind of form of bundling, we explain explicitly distortions and show that it is sometimes optimal for the monopolist to only produce one good (instead of two) or to exclude some buyers from the market. Actually, this means that the monopolist cannot fully apply his monopoly power and is better off selling both goods independently.We then define all the possible solutions in the case of a quadratic cost function for a uniform distribution of agent types and explain the implications for airlines in terms of service differentiation.

Tournut, Jacques↗
DOE OSTI2024

Next-Generation Reverse Logistics Networks of Photovoltaic Recycling: Perspectives and Challenges

With the growing adoption of solar energy as a key component of the global energy transition and its new industrial policy (the Inflation Reduction Act and others), the United States is witnessing a significant increase in solar investments. This surge in installations and domestic and reshored manufacturing of solar photovoltaic (PV) panels brings with it a pressing issue: the proper management of end-of-life (EoL) PV panels. As these panels are decommissioned, either due to reaching the end of their lifespans or due to breakage across the various stages of the forward supply chain, it becomes crucial to establish efficient reverse supply chain logistics systems to address the challenges associated with their disposal, while also unlocking the value of the inherent materials that are of critical value for other forward supply chains. This perspective article examines the challenges regarding EoL PV panels and relevant reverse supply chain and logistics networks, and proposes future research directions based on the gaps observed among academic research, industry, and policy-making challenges. We identify the main bottlenecks and hurdles including, among others, the lack of supportive regulations and absence of structured, optimized recycling infrastructure. To this end, it is proposed that the key to achieving a sustainable reverse supply chain network for solar PV panels lies in relentless end-to-end supply chain cost optimization efforts supported by enabling policies. Moreover, it is proposed that designing systematic decision-making modeling frameworks is vital for examining different possibilities and scenarios for state, region or nation-wide optimization of solar PV reverse supply chain networks. Indicative to this effect, we discuss the development of a Resource-Task-Network (RTN)-based model and demonstrate its application and benefits through a case study. We wrap up with conclusions and future research directions.

circular economy↗
DOE OSTI2021

Efficient Sampling Allocation Procedures for Optimal Quantile Selection

We propose a dynamic sampling allocation and selection paradigm for finding the alternative with the optimal quantile in a Bayesian framework. Myopic allocation policies (MAPs), analogous to existing methods in classic ranking and selection for selecting the alternative with the optimal mean, and computationally efficient selection policies are derived for selecting the alternative with the optimal quantile. Under certain conditions, we prove that the proposed MAPs and selection procedures are consistent, which means that the best quantile would be eventually correctly selected as the sample size goes to infinity. Numerical experiments demonstrate that the proposed schemes can significantly improve the performance.

Computer Science↗