Search NASA⌕ Search

SEARCH · Search NASA

Results for “Approximate dynamic programming problem”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Optimal, centralized dynamic curbside parking space zoning

In this paper we formulate a dynamic mixed integer program for optimally zoning curbside parking spaces subject to transportation policy-inspired constraints and regularization terms. First, we illustrate how given some objective of curb zoning valuation as a function of zone type (paid parking, bus stop, etc.), dynamically rezoning involves unrolling this optimization program over a fixed time horizon. Second, we implement two different solution methods given an example curb zoning valuation. In the first method, we solve long horizon dynamic zoning problems via approximate dynamic programming. In the second method, we employ Dantzig-Wolfe decomposition to break-up the mixed-integer program into a master problem and several sub-problems that can be solved in parallel. This speeds up the computational solve-time of the MIP considerably. We present simulation results and comparisons of the different employed techniques on vehicle arrival-rate data obtained for a neighborhood in downtown Seattle, Washington, USA.

Nazir, Mohammad Nawaf↗
NASA NTRS2010

Configuring Airspace Sectors with Approximate Dynamic Programming

In response to changing traffic and staffing conditions, supervisors dynamically configure airspace sectors by assigning them to control positions. A finite horizon airspace sector configuration problem models this supervisor decision. The problem is to select an airspace configuration at each time step while considering a workload cost, a reconfiguration cost, and a constraint on the number of control positions at each time step. Three algorithms for this problem are proposed and evaluated: a myopic heuristic, an exact dynamic programming algorithm, and a rollouts approximate dynamic programming algorithm. On problem instances from current operations with only dozens of possible configurations, an exact dynamic programming solution gives the optimal cost value. The rollouts algorithm achieves costs within 2% of optimal for these instances, on average. For larger problem instances that are representative of future operations and have thousands of possible configurations, excessive computation time prohibits the use of exact dynamic programming. On such problem instances, the rollouts algorithm reduces the cost achieved by the heuristic by more than 15% on average with an acceptable computation time.

Bloem, Michael↗
DOE OSTI2023

How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system

Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic programming problem, and these two have shown significant operational cost saving potentials for building energy systems. Furthermore, there is still a lack of in-depth quantitative studies on their approximation levels to the true optimality, especially in the building energy domain. To fill in the gap, this paper provides a numerical framework that enables the evaluation of the optimality levels of different controllers for building energy systems. This framework is then used to comprehensively compare the optimal control performance of both MPC and DRL controllers with given computation budgets for a single zone fan coil unit system. Note the optimality is estimated based on a user-specific selection of trade-off weights among energy costs, thermal comfort and control slew rates. Compared with the best optimality we can find through expensive optimization simulations, the best DRL agent can maximally approximate the optimality by 96.54%, which outperforms the best MPC whose optimality level is 90.11%. However, due to the stochasticity, the DRL agent is only expected to approximate the optimality by 90.42%, which is almost equivalent to the best MPC. Except for Proximal Policy Optimization (PPO), all DRL agents can have a better approximation to the optimality than the best MPC, and are expected to have better approximation than the MPC with a prediction horizon of 32 steps (15 min per step). In terms of reducing energy cost and thermal discomfort, MPC can outperform the rule-based control (RBC) by 18.47%–25.44%. DRL can be expected to outperform RBC by 18.95%–25.65% ,and the best DRL control policy can outperform RBC by 20.29%–29.72%. Although the comparison of the optimality level is performed in a perfect setting, e.g., MPC assumes perfect models, and DRL assumes a perfect offline training process and online deployment process, this can shed insight on their capabilities of approximating to the original dynamic programming problem.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

A General Framework for Bounding Approximate Dynamic Programming Schemes

For years, there has been interest in approximation methods for solving dynamic programming problems, because of the inherent complexity in computing optimal solutions characterized by Bellman’s principle of optimality. A wide range of approximate dynamic programming (ADP) methods now exists. It is of great interest to guarantee that the performance of an ADP scheme be at least some known fraction, say ß , of optimal. This letter introduces a general approach to bounding the performance of ADP methods, in this sense, in the stochastic setting. The approach is based on new results for bounding greedy solutions in string optimization problems, where one has to choose a string (ordered set) of actions to maximize an objective function. This bounding technique is inspired by submodularity theory, but submodularity is not required for establishing bounds. Instead, the bounding is based on quantifying certain notions of curvature of string functions; the smaller the curvatures the better the bound. The key insight is that any ADP scheme is a greedy scheme for some surrogate string objective function that coincides in its optimal solution and value with those of the original optimal control problem. The ADP scheme then yields to the bounding technique mentioned above, and the curvatures of the surrogate objective determine the value ß of the bound. The surrogate objective and its curvatures depend on the specific ADP.

discrete event systems↗
DOE OSTI2023

Control of Fractional Diffusion Problems via Dynamic Programming Equations

In this study, we explore the approximation of feedback control of integro-differential equations containing a fractional Laplacian term. To obtain feedback control for the state variable of this nonlocal equation, we use the Hamilton–Jacobi–Bellman equation. It is well known that this approach suffers from the curse of dimensionality, and to mitigate this problem we couple semi-Lagrangian schemes for the discretization of the dynamic programming principle with the use of Shepard approximation. This coupling enables approximation of high-dimensional problems. Numerical convergence toward the solution of the continuous problem is provided together with linear and nonlinear examples. The robustness of the method with respect to disturbances of the system is illustrated by comparisons with an open-loop control approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Experimental Validation of Approximate Dynamic Programming Based Optimization and Convergence on Microgrid Applications

Stochastic optimization can better address uncertainties in power system problems. However, when state space and action space become large, many existing approaches become computationally expensive and even infeasible. Approximate dynamic programming (ADP) attracts researchers’ attention as a powerful tool for solving power system optimization problems with reduced computational cost. In this paper, in light of the existing literature, we investigate how the ADP approach with post-decision value function approximation converges to the nearly optimal solution with improved computational speed and experimentally validate the performance of the approach for a microgrid energy optimization problem. The approximation error versus the number of iteration is studied for convergence analysis of the post-decision ADP. A flowchart is provided to illustrate the proposed ADP algorithm for a microgrid energy optimization problem. The performance of ADP and dynamic programming (DP) is compared in terms of optimization error and computational time. It has found that the post-decision ADP approach can achieve competitive optimality with improved computational speed compared to the traditional DP.

Das, Avijit↗
NASA NTRS1988

A forward method for optimal stochastic nonlinear and adaptive control

A computational approach is taken to solve the optimal nonlinear stochastic control problem. The approach is to systematically solve the stochastic dynamic programming equations forward in time, using a nested stochastic approximation technique. Although computationally intensive, this provides a straightforward numerical solution for this class of problems and provides an alternative to the usual dimensionality problem associated with solving the dynamic programming equations backward in time. It is shown that the cost degrades monotonically as the complexity of the algorithm is reduced. This provides a strategy for suboptimal control with clear performance/computation tradeoffs. A numerical study focusing on a generic optimal stochastic adaptive control example is included to demonstrate the feasibility of the method.

Bayard, David S.↗
DOE OSTI2022

Real-Time Ecodriving Control in Electrified Connected and Autonomous Vehicles Using Approximate Dynamic Programing

Connected and automated vehicles (CAVs), particularly those with a hybrid electric powertrain, have the potential to significantly improve vehicle energy savings in real-world driving conditions. In particular, the ecodriving problem seeks to design optimal speed and power usage profiles based on available information from connectivity and advanced mapping features to minimize the fuel consumption over an itinerary. This paper presents a hierarchical multilayer model predictive control (MPC) approach for improving the fuel economy of a 48 V mild-hybrid powertrain in a connected vehicle environment. Approximate dynamic programing (DP) is used to solve the receding horizon optimal control problem, whose terminal cost is approximated with the base policy obtained from the long-term optimization. The controller was tested virtually (with deterministic and Monte Carlo simulation) across multiple real-world routes, demonstrating energy savings of more than 20%. The controller was then deployed on a test vehicle equipped with a rapid prototyping embedded controller. In-vehicle testing confirm the energy savings obtained in simulation and demonstrate the real-time ability of the controller.

Automation & Control Systems↗
NASA NTRS1991

A parallel trajectory optimization tool for aerospace plane guidance

A parallel trajectory optimization algorithm is being developed. One possible mission is to provide real-time, on-line guidance for the National Aerospace Plane. The algorithm solves a discrete-time problem via the augmented Lagrangian nonlinear programming algorithm. The algorithm exploits the dynamic programming structure of the problem to achieve parallelism in calculating cost functions, gradients, constraints, Jacobians, Hessian approximations, search directions, and merit functions. Special additions to the augmented Lagrangian algorithm achieve robust convergence, achieve (almost) superlinear local convergence, and deal with constraint curvature efficiency. The algorithm can handle control and state inequality constraints such as angle-of-attack and dynamic pressure constraints. Portions of the algorithm have been tested. The nonlinear programming core algorithm performs well on a variety of static test problems and on an orbit transfer problem. The parallel search direction algorithm can reduce wall clock time by a factor of 10 for this part of the computation task.

Psiaki, Mark L.↗
NASA NTRS1990

A computational algorithm for spacecraft control and momentum management

Developments in the area of nonlinear control theory have shown how coordinate changes in the state and input spaces of a dynamical system can be used to transform certain nonlinear differential equations into equivalent linear equations. These techniques are applied to the control of a spacecraft equipped with momentum exchange devices. An optimal control problem is formulated that incorporates a nonlinear spacecraft model. An algorithm is developed for solving the optimization problem using feedback linearization to transform to an equivalent problem involving a linear dynamical constraint and a functional approximation technique to solve for the linear dynamics in terms of the control. The original problem is transformed into an unconstrained nonlinear quadratic program that yields an approximate solution to the original problem. Two examples are presented to illustrate the results.

Dzielski, John↗
NASA NTRS1976

Computational alternatives to obtain time optimal jet engine control

Two computational methods to determine an open loop time optimal control sequence for a simple single spool turbojet engine are described by a set of nonlinear differential equations. Both methods are modifications of widely accepted algorithms which can solve fixed time unconstrained optimal control problems with a free right end. Constrained problems to be considered have fixed right ends and free time. Dynamic programming is defined on a standard problem and it yields a successive approximation solution to the time optimal problem of interest. A feedback control law is obtained and it is then used to determine the corresponding open loop control sequence. The Fletcher-Reeves conjugate gradient method has been selected for adaptation to solve a nonlinear optimal control problem with state variable and control constraints.

Basso, R. J.↗
DOE OSTI2022

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Recent studies showed that reinforcement learning (RL) is a promising approach for coordination and control of distributed energy resources (DER) under uncertainties. Many existing RL approaches, including Q-learning and approximate dynamic programming, are based on lookup table methods, which become inefficient when the problem size is large and infeasible when continuous states and actions are involved. In addition, when modeling battery energy storage system (BESS), the loss of life is not reasonably considered into the decision-making process. This paper proposes an innovative deep RL method for DER coordination considering BESS degradation. The proposed deep RL is designed based on an adaptive actor-critic architecture and employs an off-policy deterministic policy gradient method for determining the dispatch operation that minimizes the operation cost and BESS life loss. Case studies were performed to validate the proposed method and demonstrate the effects of incorporating degradation models into control design.

Das, Avijit↗
NASA NTRS1994

Optimal active control for Burgers equations

A method for active fluid flow control based on control theory is discussed. Dynamic programming and fixed point successive approximations are used to accommodate the nonlinear control problem. The long-term goal of this project is to establish an effective method applicable to complex flows such as turbulence and jets. However, in this report, the method is applied to stochastic Burgers equation as an intermediate step towards this goal. Numerical results are compared with those obtained by gradient search methods.

Ikeda, Yutaka↗
DOE OSTI2020

Multistage distributionally robust mixed-integer programming with decision-dependent moment-based ambiguity sets

We study multistage distributionally robust mixed-integer programs under endogenous uncertainty, where the probability distribution of stage-wise uncertainty depends on the decisions made in previous stages. We first consider two ambiguity sets defined by decision-dependent bounds on the first and second moments of uncertain parameters and by mean and covariance matrix that exactly match decision-dependent empirical ones, respectively. For both sets, we show that the subproblem in each stage can be recast as a mixed-integer linear program (MILP). Moreover, we extend the general moment-based ambiguity set in to the multistage decision-dependent setting, and derive mixed-integer semidefinite programming (MISDP) reformulations of stage-wise subproblems. We develop methods for attaining lower and upper bounds of the optimal objective value of the multistage MISDPs, and approximate them using a series of MILPs. We deploy the Stochastic Dual Dynamic integer Programming (SDDiP) method for solving the problem under the three ambiguity sets with risk-neutral or risk-averse objective functions, and conduct numerical studies on multistage facility-location instances having diverse sizes under different parameter and uncertainty settings. Furthermore, our results show that the SDDiP quickly finds optimal solutions for moderate-sized instances under the first two ambiguity sets, and also finds good approximate bounds for the multistage MISDPs derived under the third ambiguity set. We also demonstrate the efficacy of incorporating decision-dependent distributional ambiguity in multistage decision-making processes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Solving the Dynamics-Aware Economic Dispatch Problem with the Koopman Operator

The dynamics-aware economic dispatch (DED) problem embeds low-level generator dynamics and operational constraints to enable near real-time scheduling of generation units in a power network. DED produces a more dynamic supervisory control policy than traditional economic dispatch (T-ED) that reduces overall generation costs. However, in contrast to T-ED, DED is a nonlinear, non-convex optimization problem that is computationally prohibitive to solve. We introduce a machine learning-based operator-theoretic approach for solving the DED problem efficiently. Specifically, we develop a novel discrete-time Koopman Operator (KO) formulation that embeds domain information into the structure of the KO to learn high-fidelity approximations of the generator dynamics. Using the KO approximation, the DED problem can be reformulated as a computationally tractable linear program (abbreviated DED-KO). We demonstrate the high solution quality and computational-time savings of the DED-KO model over the original DED formulation on a 9-bus test system.

King, Ethan↗
NASA NTRS1987

Robot path planning with distance-safety criterion

A method for determining an optimal path with a weighted distance-safety criterion is developed. The goal is to strike a compromise between the shortest path and the centerline path, which is safer. The method is composed of three parts: (i) construction of a region map by dividing the workspace, (ii) interregion optimization to determine the entry and departure points of the path in each region, and (iii) intraregion optimization for determining the (optimal) path segment within each region. The region map is generated by using an approximate Voronoi diagram, and region optimization is achieved using variational dynamic programming. Although developed for 2-D problems, the method can be easily extended to a class of 3-D problems. Numerical examples are presented to demonstrate the method.

Suh, Suk-Hwan↗
NASA NTRS1993

Approximate optimal guidance for the advanced launch system

A real-time guidance scheme for the problem of maximizing the payload into orbit subject to the equations of motion for a rocket over a spherical, non-rotating earth is presented. An approximate optimal launch guidance law is developed based upon an asymptotic expansion of the Hamilton - Jacobi - Bellman or dynamic programming equation. The expansion is performed in terms of a small parameter, which is used to separate the dynamics of the problem into primary and perturbation dynamics. For the zeroth-order problem the small parameter is set to zero and a closed-form solution to the zeroth-order expansion term of Hamilton - Jacobi - Bellman equation is obtained. Higher-order terms of the expansion include the effects of the neglected perturbation dynamics. These higher-order terms are determined from the solution of first-order linear partial differential equations requiring only the evaluation of quadratures. This technique is preferred as a real-time, on-line guidance scheme to alternative numerical iterative optimization schemes because of the unreliable convergence properties of these iterative guidance schemes and because the quadratures needed for the approximate optimal guidance law can be performed rapidly and by parallel processing. Even if the approximate solution is not nearly optimal, when using this technique the zeroth-order solution always provides a path which satisfies the terminal constraints. Results for two-degree-of-freedom simulations are presented for the simplified problem of flight in the equatorial plane and compared to the guidance scheme generated by the shooting method which is an iterative second-order technique.

Feeley, T. S.↗
NASA NTRS1991

Hierarchic Extensions in the Static and Dynamic Analysis of Elastic Beams

Approximate solutions of static and dynamic beam problems by the p-version of the finite element method are investigated. Within a hierarchy of engineering beam idealizations, rigorous formulations of the strain and kinetic energies for straight and circular beam elements are presented. These formulations include rotating coordinate system effects and geometric nonlinearities to allow for the evaluation of vertical axis wind turbines, the motivating problem for this research. Hierarchic finite element spaces, based on extensions of the polynomial orders used to approximate the displacement variables, are constructed. The developed models are implemented into a general purpose computer program for evaluation. Quality control procedures are examined for a diverse set of sample problems. These procedures include estimating discretization errors in energy norm and natural frequencies, performing static and dynamic equilibrium checks, observing convergence for qualities of interest, and comparison with more exacting theories and experimental data. It is demonstrated that p-extensions produce exponential rates of convergence in the approximation of strain energy and natural frequencies for the class of problems investigated.

Watson, Robert A.↗