Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

From Optimization to Sampling Through Gradient Flows

Optimization and sampling algorithms play a central role in science and engineering as they enable finding optimal predictions, policies, and recommendations, as well as expected and equilibrium states of complex systems. The notion of “optimality” is formalized by the choice of an objective function, while the notion of an “expected” state is specified by a probabilistic model for the distribution of states. Optimizing rugged objective functions and sampling multimodal distributions is computationally challenging, especially in high-dimensional problems. Here, for this reason, many optimization and sampling methods have been developed by researchers working in disparate fields such as Bayesian statistics, molecular dynamics, genetics, quantum chemistry, machine learning, weather forecasting, econometrics, and medical imaging.

Trillos, N. García↗

Potential State Regulatory Pathways to Facilitate Low-Carbon Fuels

States and the federal government are increasingly engaged in the challenges around decarbonizing the electric grid. In particular, regulators, consumers, stakeholders, and utilities recognize the need to carefully consider the role natural gas will play in a decarbonized future. A variety of technology and policy options to reduce greenhouse gas emissions associated with natural gas use are available, including energy efficiency programs, demand reduction tools, strategic electrification, and strategies to reduce emissions from natural gas production, transportation, and consumption. Low-carbon fuels – mainly renewable natural gas (RNG) and clean hydrogen – are being considered an important component of decarbonization goals. RNG and hydrogen may be able to meaningfully reduce emissions from processes independent of geologic natural gas, displacing emissions of methane, a powerful greenhouse gas. Although RNG and hydrogen are not cost-competitive today with geologic natural gas and are smaller in scale and potential than other decarbonization options, they can be explored as potential critical tools to decarbonize sectors that are difficult to electrify or shift off of natural gas entirely, such as air travel, industrial processes, maritime transport, long-distance trucking, space heating on cold days, and railroads (Nadel, 2022). The role of this report is to provide informational context for state utility regulators to understand the impacts of and challenges associated with broader integration of low-carbon fuels, followed by examples of state regulatory actions taken to date to facilitate the development of low-carbon fuels. Setting clear guidance to calculate the environmental benefits of low-carbon fuels and continuing federal and state investments in research and development to reduce costs relative to fossil fuels will be important steps to take to signal the desire to grow the market for these fuels. State public utility commissions may play a key role in setting regulatory frameworks for low-carbon fuels and ensuring that ratepayer funds, if utilized, are done so to further the public interest. This report is intended to summarize decisions that states have made to date on low-carbon fuels. In the spirit of understanding the current market and sharing information, this report provides success stories, and lessons learned across states as regulators implement varying strategies to achieve decarbonization objectives while maintaining their focus on affordability, safety, and reliability of the energy system. The report begins with an introduction of the role of natural gas in the U.S. economy (Section I) and background information on natural gas use, decarbonization, and low-carbon fuels (Section II). Next, the report describes the current market by discussing the scale of current production, emissions intensity, resource potential, and costs of low-carbon fuels compared to geologic natural gas (Section III). Following these sections, the report describes four strategies states have employed to facilitate low-carbon fuels: opening exploratory dockets, approving voluntary tariffs for customers, approving interconnection tariffs for producers, and considering portfolio-wide procurement targets (Section IV). This section lists states that have taken actions in each category, citing utility filings, commission decisions, stakeholder comments, and other relevant sources. Finally, the report concludes with suggested questions regulators may wish to consider regarding low-carbon fuels, in the interest of preparing to make decisions in the future (Section V). These questions include: Are there existing regulatory or technical barriers to voluntary purchases of low-carbon fuels? Can customers work with utilities to procure low-carbon fuels; are producers able to interconnect projects without significant barriers to entry? Should the infrastructure and/or commodity costs of low-carbon fuels be socialized among all ratepayers, or borne solely by the large commercial and industrial (C&I) customers currently driving the market? Should regulated natural gas and/or electric utilities own and operate low-carbon fuel production? How should regulators consider the unique decarbonization potential of low-carbon fuels, particularly for hard-to-abate sectors, in decision-making? Is additional direction or clarity from state policymakers needed? What no-regrets approaches can help facilitate both near-term RNG development and long-term development of hydrogen and other zero-carbon fuels? We collectively wish to express our gratitude to the U.S. Department of Energy, Office of Fossil Energy and Carbon Management, for supporting this report and other technical assistance resources for state regulators on natural gas topics. State regulators operate under a variety of policy environments, and states have vastly different types of energy resources, infrastructure, and customers. While there is no optimal regulatory, policy, or technological solution that will be successful in every state, state regulators can benefit by exchanging lessons learned with their peers across the country. We look forward to continued engagement with our fellow commissioners, commission staff, NARUC, the U.S. Department of Energy, and other stakeholders to develop sound regulation in the public interest.

03 NATURAL GAS↗

DTS: Building custom, intelligent schedulers

DTS is a decision-theoretic scheduler, built on top of a flexible toolkit -- this paper focuses on how the toolkit might be reused in future NASA mission schedulers. The toolkit includes a user-customizable scheduling interface, and a 'Just-For-You' optimization engine. The customizable interface is built on two metaphors: objects and dynamic graphs. Objects help to structure problem specifications and related data, while dynamic graphs simplify the specification of graphical schedule editors (such as Gantt charts). The interface can be used with any 'back-end' scheduler, through dynamically-loaded code, interprocess communication, or a shared database. The 'Just-For-You' optimization engine includes user-specific utility functions, automatically compiled heuristic evaluations, and a postprocessing facility for enforcing scheduling policies. The optimization engine is based on BPS, the Bayesian Problem-Solver (1,2), which introduced a similar approach to solving single-agent and adversarial graph search problems.

Hansson, Othar↗

Incorporating Operational Uncertainties into the Dispatch of an Integrated Solar and Storage System

The economic assessment of hybrid energy systems (HES) pairing battery energy storage systems (BESSs) and photovoltaics (PV) is highly important for advancing their deployment in power systems. This paper presents an innovative assessment framework, including an optimal control policy for dispatch under uncertainty and procedures for exploring control parameters that maximize economic benefits. The proposed dispatch policy consists of two steps using system forecast information. The first step is to determine whether a BESS will be used within an operational scheduling time frame based on the probability of events and their thresholds. Once the dispatch of BESS is triggered, a model predictive control (MPC) is carried out in the second step for scheduling using the expected value of system information. By exercising this policy with different thresholds, one can explore the trade-offs between short-term benefits and battery lifetime, and identify an optimal threshold that maximizes the total economic benefits within the battery lifetime. An evaluation study in a real-world HES project is presented to illustrate the proposed framework. Compared with traditional optimal dispatch algorithms, the proposed method can significantly improve the economic benefits of an HES scheduled under forecast uncertainties.

Ma, Xu↗

Combined optimal control and estimation.

Combined optimization problem, equivalent to dual control problem, considering determination of optimal control policies for plant under random disturbances, using iterative equations

CONTROL SYSTEM↗

Sensitivity of joint contagiousness and susceptibility-based dynamic optimal control strategies for HIV prevention

Predicting the population-level effects of an infectious disease intervention that incorporate multiple modes of intervention is complicated by the joint non-linear dynamics of both infection transmission and the intervention itself. In this paper, we consider the sensitivity of Dynamic Optimal Control Profiles (DOCPs) for the optimal joint investment in both a contagiousness and susceptibility-based control of HIV to bio-behavioral, economic, and programmatic assumptions. The DOCP is calculated using recently developed numerical algorithms that allow controls to be represented by a set of piecewise constant functions that maintain a constant yearly budget. Our transmission model assumes multiple stages of HIV infection corresponding to acute and chronic infection and both within- and between-individual behavioral heterogeneity. We parameterize a baseline scenario from a longitudinal study of sexual behavior in MSM and consider sensitivity of the DOCPs to deviations from that baseline scenario. In the baseline scenario, the primary determinant of the dominant control were programmatic factors, regardless of budget. In sensitivity analyses, the qualitative aspects of the optimal control policy were often robust to significant deviation in assumptions regarding transmission dynamics. In addition, we found several conditions in which long-term joint investment in both interventions was optimal. Our results suggest that modeling in the service of decision support for intervention design can improve population-level effects of a limited set of economic resources. We found that economic and programmatic factors were as important as the inherent transmission dynamics in determining population-level intervention effects. Given our finding that the DOCPs were robust to alternative biological and behavioral assumptions it may be possible to identify DOCPs even when the data are not sufficient to identify a transmission model.

59 BASIC BIOLOGICAL SCIENCES↗

Guidance analysis of the aeroglide plane change maneuver as a turning point problem

The development of guidance approximations for the atmospheric (aeroglide) portion of the minimum fuel, orbital plane change, trajectory optimization problem is described. Asymptotic methods are used to reduce the two point, boundary value, optimization problem to a turning point problem from the bank angle control. The turning point problem solution, which yields an approximate optimal control policy, is given in terms of parabolic cylinder functions, which are tabulated, and integral expressions, which must be numerically computed. Comparisons of the former, over their region of validity, with optimal control solutions show good qualitative agreement. Additional work and analysis is needed to compute the guidance approximation work.

Gracey, Christopher↗

Optimal startup control of a jacketed tubular reactor.

The optimal startup policy of a jacketed tubular reactor, in which a first-order, reversible, exothermic reaction takes place, is presented. A distributed maximum principle is presented for determining weak necessary conditions for optimality of a diffusional distributed parameter system. A numerical technique is developed for practical implementation of the distributed maximum principle. This involves the sequential solution of the state and adjoint equations, in conjunction with a functional gradient technique for iteratively improving the control function.

Hahn, D. R.↗

Controlling colloidal crystals via morphing energy landscapes and reinforcement learning

We report a feedback control method to remove grain boundaries and produce circular shaped colloidal crystals using morphing energy landscapes and reinforcement learning–based policies. We demonstrate this approach in optical microscopy and computer simulation experiments for colloidal particles in ac electric fields. First, we discover how tunable energy landscape shapes and orientations enhance grain boundary motion and crystal morphology relaxation. Next, reinforcement learning is used to develop an optimized control policy to actuate morphing energy landscapes to produce defect-free crystals orders of magnitude faster than natural relaxation times. Morphing energy landscapes mechanistically enable rapid crystal repair via anisotropic stresses to control defect and shape relaxation without melting. This method is scalable for up to at least N = 10 3 particles with mean process times scaling as N 0.5 . Further scalability is possible by controlling parallel local energy landscapes (e.g., periodic landscapes) to generate large-scale global defect-free hierarchical structures.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Stochastic ordering properties and optimal routing control for a class of finite capacity queueing systems

The problem of routing jobs to parallel queues with identical exponential servers and unequal finite buffer capacities is considered. Stochastic ordering and weak majorization properties on critical performance measures are established by means of event-driven inductions. In particular, it is shown that the intuitive 'join the shortest non-full queue' (SNQ) policy is optimal with respect to an overall function that accounts for holding and blocking costs. Moreover, the buffer allocation problem is solved by proving the intuitive result that, for a fixed total buffer capacity, the optimal allocation scheme is the one in which the difference between the maximum and minimum queue capacities is minimized, i.e., becomes either 0 or 1.

Towsley, Don↗

Optimal Control with Restrictions for a Diffusion Risk Model Under Constant Interest Force

In this paper, we study optimal dividend problems in a diffusion risk model for two different cases depending on whether reinsurance is incorporated. In either case, the dividend rate is bounded above by a constant, and the company earns investment income at a constant force of interest. Unlike existing approaches in the literature dealing with optimal problems with interest, we allow the force of interest to be greater than the discount factor, and we use a different method to solve the corresponding Hamilton–Jacobi–Bellman (HJB) equation instead of introducing a confluent hypergeometric function. We conclude that the optimal dividend policy is of a threshold type and show that the corresponding dividend barrier is nondecreasing in the dividend rate bound. In cases where there is no reinsurance, we construct an auxiliary reflecting control problem to find the nonzero dividend barrier. If proportional reinsurance is purchased, the optimal reinsurance strategy looks somewhat strange. The optimal retention level of risk first increases monotonically with risk reserve to some possible value (less than 1) and then stays at level 1 for a while or, if 1 has been reached, finally, it decreases to 0.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Optimal Limited Contingency Planning

For a given problem, the optimal Markov policy over a finite horizon is a conditional plan containing a potentially large number of branches. However, there are applications where it is desirable to strictly limit the number of decision points and branches in a plan. This raises the question of how one goes about finding optimal plans containing only a limited number of branches. In this paper, we present an any-time algorithm for optimal k-contingency planning. It is the first optimal algorithm for limited contingency planning that is not an explicit enumeration of possible contingent plans. By modelling the problem as a partially observable Markov decision process, it implements the Bellman optimality principle and prunes the solution space. We present experimental results of applying this algorithm to some simple test cases.

Meuleau, Nicolas↗

A Multiobjective Optimization Framework for Stochastic Control of Complex Systems

This paper addresses the problem of minimizing the long-run expected average cost of a complex system consisting of subsystems that interact with each other and the environment. We treat the stochastic control problem as a multiobjective optimization problem of the one-stage expected costs of the subsystems, and we show that the control policy yielding the Pareto optimal solution is an optimal control policy that minimizes the average cost criterion for the entire system. For practical situations with constraints consistent to those we study here, our results imply that the Pareto control policy may be of value in deriving online an optimal control policy in complex systems.

Complex systems↗

Multiobjective optimization to a TB-HIV/AIDS coinfection optimal control problem

We consider a recent coinfection model for Tuberculosis (TB), Human Immunodeficiency Virus (HIV) infection, and Acquired Immunodeficiency Syndrome (AIDS) proposed in Silva and Torres (Discr Contin Dyn Syst 35(9):4639–4663, 2015). We introduce and analyze a multiobjective formulation of an optimal control problem, where the two conflicting objectives are minimization of the number of HIV-infected individuals with AIDS clinical symptoms and coinfected with AIDS and active TB; and costs related to prevention and treatment of HIV and/or TB measures. The proposed approach eliminates some limitations of previous works. The results of the numerical study provide comprehensive insights about the optimal treatment policies and the population dynamics resulting from their implementation. Some nonintuitive conclusions are drawn. Overall, the simulation results demonstrate the usefulness and validity of the proposed approach.

97 MATHEMATICS AND COMPUTING↗

Online data-enabled predictive control

We develop an online data-enabled predictive (ODeePC) control method for optimal control of unknown systems, building on the recently proposed DeePC (Coulson et al., 2019). Our proposed ODeePC method leverages a primal-dual algorithm with real-time measurement feedback to iteratively compute the corresponding real-time optimal control policy as system conditions change. The proposed ODeePC conceptual-wise resembles standard adaptive system identification and model predictive control (MPC), but it provides a new alternative for the standard methods. ODeePC is enabled by computationally efficient methods that exploit the special structure of the Hankel matrices in the context of DeePC with Fast Fourier Transform (FFT) and primal-dual algorithm We provide theoretical guarantees regarding the asymptotic behavior of ODeePC, and we demonstrate its performance through numerical examples.

97 MATHEMATICS AND COMPUTING↗

Online Data-Enabled Predictive Control

We develop an online data-enabled predictive (ODeePC) control method for trajectory tracking of unknown systems, building upon the recently proposed DeePC. Our proposed ODeePC method leverages a primal-dual algorithm with real-time measurement feedback to iteratively compute the corresponding real-time optimal control policy as system conditions change. Specifically, our developed ODeePC: a) records data from the unknown system and updates the underlying primal-dual algorithm dynamically, b) can track changes in the system's operating point and adjust the control inputs, and c) is computationally efficient as it deploys a Fast Fourier Transform-based algorithm enabling the fast computation of the product of a non-square Hankel matrix with a vector. We provide theoretical guarantees regarding the asymptotic behavior of ODeePC and demonstrate its performance through a power system application.

61 RADIATION PROTECTION AND DOSIMETRY↗

Preliminary Work for Examining the Scalability of Reinforcement Learning

Researchers began studying automated agents that learn to perform multiple-step tasks early in the history of artificial intelligence (Samuel, 1963; Samuel, 1967; Waterman, 1970; Fikes, Hart & Nilsonn, 1972). Multiple-step tasks are tasks that can only be solved via a sequence of decisions, such as control problems, robotics problems, classic problem-solving, and game-playing. The objective of agents attempting to learn such tasks is to use the resources they have available in order to become more proficient at the tasks. In particular, each agent attempts to develop a good policy, a mapping from states to actions, that allows it to select actions that optimize a measure of its performance on the task; for example, reducing the number of steps necessary to complete the task successfully. Our study focuses on reinforcement learning, a set of learning techniques where the learner performs trial-and-error experiments in the task and adapts its policy based on the outcome of those experiments. Much of the work in reinforcement learning has focused on a particular, simple representation, where every problem state is represented explicitly in a table, and associated with each state are the actions that can be chosen in that state. A major advantage of this table lookup representation is that one can prove that certain reinforcement learning techniques will develop an optimal policy for the current task. The drawback is that the representation limits the application of reinforcement learning to multiple-step tasks with relatively small state-spaces. There has been a little theoretical work that proves that convergence to optimal solutions can be obtained when using generalization structures, but the structures are quite simple. The theory says little about complex structures, such as multi-layer, feedforward artificial neural networks (Rumelhart & McClelland, 1986), but empirical results indicate that the use of reinforcement learning with such structures is promising. These empirical results make no theoretical claims, nor compare the policies produced to optimal policies. A goal of our work is to be able to make the comparison between an optimal policy and one stored in an artificial neural network. A difficulty of performing such a study is finding a multiple-step task that is small enough that one can find an optimal policy using table lookup, yet large enough that, for practical purposes, an artificial neural network is really required. We have identified a limited form of the game OTHELLO as satisfying these requirements. The work we report here is in the very preliminary stages of research, but this paper provides background for the problem being studied and a description of our initial approach to examining the problem. In the remainder of this paper, we first describe reinforcement learning in more detail. Next, we present the game OTHELLO. Finally we argue that a restricted form of the game meets the requirements of our study, and describe our preliminary approach to finding an optimal solution to the problem.

Clouse, Jeff↗

Synchronization-Avoiding Graph Algorithms

Because they were developed for optimal sequential complexity, classical graph algorithms as found in textbooks have strictly-defined orders of operations. Enforcing a prescribed order of operations, or even an approximate order, in a distributed memory setting requires significant amounts of synchronization, which in turn can severely limit scalability. As a result, new algorithms are typically required to achieve scalable performance, even for solving well-known graph problems. Yet, even in these cases, parallel graph algorithms are written according to parallel programming models that evolved for, e.g., scientific computing, and that still have inherent, and scalability-limiting, amounts of synchronization. In this paper we present a new approach to parallel graph algorithms: synchronization-avoiding algorithms. To eliminate synchronization and its associated overhead, synchronization-avoiding algorithms perform work in an unordered and fully asynchronous fashion in such a way that the result is constantly refined toward its final state. “Wasted” work is minimized by locally prioritizing tasks using problem-dependent task utility metrics. We classify algorithms for graph applications into two broad categories: algorithms with monotonic updates (which evince global synchronization) and algorithms with nonmonotonic updates (which evince vertex-centric synchronization). We apply our approach to both classes and develop novel, synchronization-avoiding algorithms for solving exemplar problems: SSSP and connected components for the former, graph coloring for the latter. We demonstrate that eliminating synchronization in conjunction with effective scheduling policies and optimizations in the runtime results in improved scalability for both classes of algorithms.

Firoz, Jesun S.↗