SEARCH · Search NASA
Results for “Dynamic Programming”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The Heuristic Dynamic Programming Approach in Boost Converters
In this study, a heuristic dynamic programming controller is proposed to control a boost converter. Conventional controllers such as proportional-integral-derivative (PID) or proportional-integral (PI) are designed based on the linearized small-signal model near the operating point. Therefore, the performance of the controller during the start-up, the load change, or the input voltage variation is not optimal since the system model changes by varying the operating point. The heuristic dynamic programming controller optimally controls the boost converter by following the approximate dynamic programming. The advantage of the HDP is that the neural network-based characteristic of the proposed controller enables boost converters to easily cope with large disturbances. An HDP with a well-trained critic and action networks can perform as an optimal controller for the boost converter. To compare the effectiveness of the traditional PI-based and the HDP boost converter, the simulation results are provided.
Exponential Decay in the Sensitivity Analysis of Nonlinear Dynamic Programming
In this report, we study the sensitivity of discrete-time dynamic programs with nonlinear dynamics and objective to perturbations in the initial conditions and reference parameters. Under uniform controllability and boundedness assumptions for the problem data, we prove that the directional derivative of the optimal state and control at time $k$, $x^*_k$, and $u^*_k$, with respect to the reference signal at time $i$, $d_i$, will have exponential decay in terms of $|k-i|$ with a decay rate $\rho$ independent of the temporal horizon length. The key technical step is to prove that a version of the convexification approach proposed by Verschueren et al. can be applied to the KKT conditions and results in a convex quadratic program with uniformly bounded data. In turn, Riccati techniques can be further employed to obtain the sensitivity result, borne from the observation that the directional derivatives are solutions of quadratic programs with structure similar to the KKT conditions themselves. We validate our findings with numerical experiments on a small nonlinear, nonconvex, dynamic program.
Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators
In this paper a neural network heuristic dynamic programing (HDP) is used for optimal control of the virtual inertia-based control of grid connected three-phase inverters. It is shown that the conventional virtual inertia controllers are not suited for non-inductive grids. A neural network-based controller is proposed to adapt to any impedance angle. Applying an adaptive dynamic programming controller instead of a supervised controlled method enables the system to adjust itself to different conditions. The proposed HDP consists of two subnetworks: critic network and action network. These networks can be trained during the same training cycle to decrease the training time. The simulation results confirm that the proposed neural network HDP controller performs better than the traditional direct-fed voltage and/or reactive power controllers in virtual inertia control schemes.
A General Framework for Bounding Approximate Dynamic Programming Schemes
For years, there has been interest in approximation methods for solving dynamic programming problems, because of the inherent complexity in computing optimal solutions characterized by Bellman’s principle of optimality. A wide range of approximate dynamic programming (ADP) methods now exists. It is of great interest to guarantee that the performance of an ADP scheme be at least some known fraction, say ß , of optimal. This letter introduces a general approach to bounding the performance of ADP methods, in this sense, in the stochastic setting. The approach is based on new results for bounding greedy solutions in string optimization problems, where one has to choose a string (ordered set) of actions to maximize an objective function. This bounding technique is inspired by submodularity theory, but submodularity is not required for establishing bounds. Instead, the bounding is based on quantifying certain notions of curvature of string functions; the smaller the curvatures the better the bound. The key insight is that any ADP scheme is a greedy scheme for some surrogate string objective function that coincides in its optimal solution and value with those of the original optimal control problem. The ADP scheme then yields to the bounding technique mentioned above, and the curvatures of the surrogate objective determine the value ß of the bound. The surrogate objective and its curvatures depend on the specific ADP.
An (adjustable RN)-based multi-OLT PON and the dynamic programming algorithm for downstream channel
In this paper, a multiple optical line terminal (OLT) passive optical network (PON) and the dynamic programming algorithm for downstream channel is proposed. By wavelength-selectable module, the adjustable remote node (RN) is constructed. By this adjustable RN, a simple and efficient multi-OLT PON is designed. By the way of first-plan and then-load, optical network unit (ONU) groups are established in order to realize the fast loading of ONUs. Specially, by introducing dynamic programming algorithm, the wavelength bandwidth utilization is maximized. Finally, by the simulation and analysis, the effectiveness of the proposed scheme is demonstrated.
A Fast Temporal Decomposition Procedure for Long-Horizon Nonlinear Dynamic Programming
We propose a fast temporal decomposition procedure for solving long-horizon nonlinear dynamic programs. The core of the procedure is sequential quadratic programming (SQP) that utilizes a differentiable exact augmented Lagrangian as the merit function. Within each SQP iteration, we approximately solve the Newton system using an overlapping temporal decomposition strategy. We show that the approximate search direction is still a descent direction of the augmented Lagrangian provided the overlap size and penalty parameters are suitably chosen, which allows us to establish the global convergence. Moreover, we show that a unit step size is accepted locally for the approximate search direction and further establish a uniform, local linear convergence over stages. This local convergence rate matches the rate of the recent Schwarz scheme (Na et al. 2022). However, the Schwarz scheme has to solve nonlinear subproblems to optimality in each iteration, whereas we only perform a single Newton step instead. Numerical experiments validate our theories and demonstrate the superiority of our method.
Experimental Validation of Approximate Dynamic Programming Based Optimization and Convergence on Microgrid Applications
Stochastic optimization can better address uncertainties in power system problems. However, when state space and action space become large, many existing approaches become computationally expensive and even infeasible. Approximate dynamic programming (ADP) attracts researchers’ attention as a powerful tool for solving power system optimization problems with reduced computational cost. In this paper, in light of the existing literature, we investigate how the ADP approach with post-decision value function approximation converges to the nearly optimal solution with improved computational speed and experimentally validate the performance of the approach for a microgrid energy optimization problem. The approximation error versus the number of iteration is studied for convergence analysis of the post-decision ADP. A flowchart is provided to illustrate the proposed ADP algorithm for a microgrid energy optimization problem. The performance of ADP and dynamic programming (DP) is compared in terms of optimization error and computational time. It has found that the post-decision ADP approach can achieve competitive optimality with improved computational speed compared to the traditional DP.
Solving Stochastic Dynamic Programming Problems: A Mixed Complementarity Approach
We detail a mixed complementarity problem (MCP) formulation of continuous state dynamic programming problems (DP-MCP). We write the solution to projection methods in value function iteration (VFI) as a joint set of optimality conditions that characterize maximization of the Bellman equation; and approximation of the value function. The MCP approach replaces the iterative component of projection based VFI with a one-shot solution to a square system of complementary conditions. We provide three numerical examples to illustrate our approach.
Approximate Dynamic Programming With Enhanced Off-Policy Learning for Coordinating Distributed Energy Resources
Herein this paper proposes an innovative approximate dynamic programming (ADP) method for distributed energy resource coordination with the loss of life of battery energy storage system (BESS) explicitly modeled. The dispatch policy is designed to account for both calendrical and cyclical aging effects on BESS, explicitly modeling the impacts of ambient temperature on BESS lifespan. The proposed ADP employs an adaptive critic method and enhanced off-policy deterministic policy gradient (DPG) strategy, addressing the limitations of the on-policy gradient-based ADP approaches, including inadequate exploration, low data usage, and computational complexity. In particular, a customized policy is proposed to guide the algorithm to explore some promising decisions and thereby improve exploration capability and learning efficiency compared to conventional DPG-based learning approaches, which may struggle to find a global optimum due to random noisy action-based exploration or require expert demonstration with extra effort. The proposed method is illustrated using the IEEE 123-node system and compared with the existing ADP methods to prove solution accuracy and demonstrate the effects of incorporating degradation models into control design. Case studies showed that the proposed ADP effectively coordinates DERs with a 10 times smaller optimization gap compared to existing methods, and the incorporation of the BESS life loss model ensures the expected lifespan and results in significant cost savings.
Dual heuristic dynamic programing control of grid-connected synchronverters
A new approach to control a grid-connected synchronverter by using a dual heuristic dynamic programing (DHP) design is presented. The disadvantages of conventional synchronverter controller such as the challenges to cope with nonlinearity, uncertainties, and non-inductive grids are discussed. To deal with the aforementioned challenges a neural network–based adaptive critic design is introduced to optimize the associated cost function. The characteristic of the neural networks facilitates the performance under uncertainties and unknown parameters (e.g. different power angles). The proposed DHP design includes three neural networks: system NN, action NN, and critic NN. The simulation results compare the performance of the proposed DHP with a traditional PI-based design and with a neural network predictive controller. It is shown a well-trained DHP design performs in a trajectory, which is more optimal compared to the other two controllers.
Optimal scheduling for profit maximization of energy storage merchants considering market impact based on dynamic programming
This paper analyzes how electricity merchants' market impact affects merchants' profit. Energy storage has long been studied for its role in maximizing profit, and merchant decisions are assumed to have no impact on market prices. However, the trading decisions of large-scale energy storage merchants (e.g., pumped storage hydro) will affect the market prices. This paper employs dynamic programming theory to investigate merchants' optimal economic dispatch considering the market impact and physical characteristics of storage systems. Our findings show that the State-of-Charge (SOC) based analytical solution significantly facilitates energy storage merchants' decision-making. The SOC range is segmented into three regions by two optimal SOC reference points, which depend on the available energy in storage, given prices, and market impact. By comparing the current storage SOC with the reference points, the merchant can get the corresponding optimal actions. We analytically show that if the merchant neglects the market impact on the power market, she will exaggerate her expected profit when the price-taker and price-maker merchants have the same generating and pumping upper limits offered to Independent System Operators (ISOs). Furthermore, the profit-maximizing merchant must, therefore, assay to balance the trade-off correctly between the intensity of market impact and the dispatched power. Our findings are verified by numerical simulation, and results demonstrate the ramifications for electricity merchants in energy arbitrage decisions.
Eco-driving Profile Optimization by Dynamic Programming for Battery Electric Vehicles
Although full automation has not yet been achieved, automated vehicles are a valid research area. Not only would automated vehicles provide ultimate driver convenience, but they would maximize energy efficiency by eliminating undesired human driving behaviors and optimally controlling the powertrain. From the perspective of control related to energy saving, speed profile optimization is important for improving system efficiency and satisfying passenger demands. This study employs Dynamic Programming (DP) to solve the constrained optimal problem for travel time, distance, and speed limit by exploring all possible control options. The solutions obtained by DP demonstrate consistent control patterns combining four control modes-acceleration, cruising, coasting, and braking, with cruising or coasting being selective depending on the boundary conditions. Further, this study introduces DP-based simulation results and attempts to provide comprehensive interpretations of the optimal policy by analyzing the essential factors that affect the control problem, including boundary conditions, road load, and powertrain characteristics. Based on these interpretations, the control concepts can be explained as the optimal policy selecting the best control option based on system efficiency and boundary conditions. The results of DP are compared with a human-like driver model to show that the optimal speed profiles can effectively reduce energy consumption.
ddcMD: A fully GPU-accelerated molecular dynamics program for the Martini force field
We have implemented the Martini force field within Lawrence Livermore National Laboratory’s molecular dynamics program, ddcMD. The program is extended to a heterogeneous programming model so that it can exploit graphics processing unit (GPU) accelerators. In addition to the Martini force field being ported to the GPU, the entire integration step, including thermostat, barostat, and constraint solver, is ported as well, which speeds up the simulations to 278-fold using one GPU vs one central processing unit (CPU) core. A benchmark study is performed with several test cases, comparing ddcMD and GROMACS Martini simulations. The average performance of ddcMD for a protein–lipid simulation system of 136k particles achieves 1.04 µs/day on one NVIDIA V100 GPU and aggregates 6.19 µs/day on one Summit node with six GPUs. The GPU implementation in ddcMD offloads all computations to the GPU and only requires one CPU core per simulation to manage the inputs and outputs, freeing up remaining CPU resources on the compute node for alternative tasks often required in complex simulation campaigns.
Accumulator Sizing for the Hybrid Hydraulic Electric Architecture (HHEA) Using Dynamic Programming
Abstract The Hybrid Hydraulic Electric Architecture (HHEA) is an approach to electrifying and improving that does not require very large electric machines. The majority power is provided hydraulically via a set of common pressure rails while small electric machines are used to modulate that power to meet exact demand. Each common pressure rail has an accumulator, which keeps the pressure of each rail near constant as long as the volume in the accumulator does not change much. A main pump adds or removes fluid from each accumulator as needed to reduce volume changes in the accumulators. It is the operation of this main pump which is investigated here. In previous analysis it was assumed that the size of the accumulators was large enough to account for any difference in fluid volume without significantly changing the pressure. In this paper, the required accumulator sizes are studied using a dynamic programming approach. At each time step the inlet and outlet of the main pump could be connected to any of the common pressure rails. These decisions for the main pump were optimized to minimize the required accumulator size. Case studies are conducted using drive cycles from a 22-ton excavator, a 20-ton wheel loader and a 5-ton excavator. The machines could be operated with total accumulator sizes of less than 20, 10, and 5 liters respectively. Required accumulator size was found to vary with pump flow rate, the frequency at which the pump/motor was able to change operating conditions, the tolerance level of pressure deviation and the accumulator’s polytropic index.
Control of Fractional Diffusion Problems via Dynamic Programming Equations
In this study, we explore the approximation of feedback control of integro-differential equations containing a fractional Laplacian term. To obtain feedback control for the state variable of this nonlocal equation, we use the Hamilton–Jacobi–Bellman equation. It is well known that this approach suffers from the curse of dimensionality, and to mitigate this problem we couple semi-Lagrangian schemes for the discretization of the dynamic programming principle with the use of Shepard approximation. This coupling enables approximation of high-dimensional problems. Numerical convergence toward the solution of the continuous problem is provided together with linear and nonlinear examples. The robustness of the method with respect to disturbances of the system is illustrated by comparisons with an open-loop control approach.
Route-Optimized Energy Management of Connected and Automated Multi-Mode Plug-In Hybrid Electric Vehicle Using Dynamic Programming
This paper presents a methodology to optimize the blending of charge-depleting (CD) and charge-sustaining (CS) modes in a multi-mode plug-in hybrid electric vehicle (PHEV) that reduces overall energy consumption when the selected route cannot be completely driven in all-electric mode. The PHEV used in this investigation is the second-generation Chevrolet Volt and as many as four instrumented vehicles were utilized simultaneously on road to acquire validation data. The optimization method used is dynamic programming (DP) paired with a reduced-order powertrain model to enable onboard embedded controller compatibility and computational efficiency in optimally blending CD, CS modes over the entire drive route. The objective of the optimizer is to enable future Connected and Automated Vehicles (CAVs) to best utilize onboard energy for minimum overall energy consumption based on speed and elevation profile information from Intelligent Transportation Systems (ITS), Internet of Things (IoT), High-definition Mapping, and onboard sensing technologies. Emphasis is placed on runtime minimization to quickly react and plan an optimal mode scheme in highly dynamic road conditions with minimal computational resources. On-road performance of the optimizer paired with automated CD and CS mode selection is evaluated on a fleet of four instrumented Chevrolet Volts in a variety of driving scenarios. Here, the results indicate variable energy savings depending on the drive route and initial battery SOC with potential ranging between 2 to 12% and less than 7 seconds initial optimization for a 24-mile drive cycle.
Comparison of Optimal Energy Management Strategies Using Dynamic Programming, Model Predictive Control, and Constant Velocity Prediction
Due to the recent advancements in autonomous vehicle technology, future vehicle velocity predictions are becoming more robust which allows fuel economy (FE) improvements in hybrid electric vehicles through optimal energy management strategies (EMS). A realworld highway drive cycle (DC) and a controls-oriented 2017 Toyota Prius Prime model are used to study potential FE improvements. We proposed three important metrics for comparison: (1) perfect full drive cycle prediction using dynamic programming, (2) 10-second prediction horizon model predictive control (MPC), and (3) 10-second constant velocity prediction. These different velocity predictions are put into an optimal EMS derivation algorithm to derive optimal engine torque and engine speed. The results show that the constant velocity prediction algorithm outperformed the baseline control strategy but underperformed the MPC strategy with an average 1.58% and 2.45% of FE improvement with highway and city-highway DC. Also, using a 10-second prediction window MPC strategy provided FE improvement results close to the full drive cycle prediction case. MPC has the potential to achieve 60%-65% and 70% - 80% of global FE improvement over highway and city-highway DC respectively.