Mathematical programming and the control of Markov chains
Linear and dynamic programming and Markov chain optimal control
Kleinman, A. J.↗
SEARCH · Search NASA
Results for “controlled Markov chains”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A Multiobjective Optimization Framework for Online Stochastic Optimal Control in Hybrid Electric Vehicles
The increasing urgency to extract additional efficiency from hybrid propulsion systems has led to the development of advanced power management control algorithms. In this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain and we show that the control policy yielding the Pareto optimal solution minimizes online the long-run expected average cost per unit time criterion. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion. Both solutions achieved the same cumulative fuel consumption demonstrating that the online Pareto control policy is an optimal control policy.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
The Control of Nonlinear Stochastic Control Systems Under Discounted Performance Criteria
Finite Markov chain control and numerical algorithm for optimal control
Harris, C. A.↗
Centralized Stochastic Optimal Control of Complex Systems
In this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain using the long-run expected average cost per unit time criterion, and we show that the control policy yielding the Pareto optimal solution minimizes the average cost criterion online. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion.
complex systems↗
Pareto Efficient Policy for Supervisory Power Management Control
n this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV opera- tion as a controlled Markov chain using the long-run expected average cost per unit time criterion, and we show that the control policy yielding the Pareto optimal solution minimizes the average cost criterion online. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion.
Markov chain↗
Constructing the Pareto front for multi-objective Markov chains handling a strong Pareto policy approach
In this paper, we present a multi-objective solution of the Pareto front for a certain class of discrete-time ergodic controllable Markov chains. For solving the problem, we transform the original multi-objective problem into an equivalent nonlinear programming problem implementing the Lagrange principle. We then propose to adopt the Tikhonov’s regularization method to ensure the convergence of the cost functions to a unique point into the Pareto front. We show that Pareto policies are characterized as optimal policies. One of the fundamental problems related to the construction of the Pareto front is the existence and characterization of both, Pareto and strong Pareto policies. By introducing the Tikhonov’s regularizator, we ensure the existence of strong Pareto policies. This paper proposes a real solution to this problem. We formulate the original problem considering several constraints: (a) we employ the c-variable method for the introduction of linear constraints over the nonlinear problem and (b) restrict the cost functions, allowing points in the Pareto front to have a small distance from one another. The constraints imposed by the c-variable method make the problem computationally tractable and the restriction imposed by the small distance change ensures the continuation of the Pareto front. The resulting equation in this nonlinear system is an optimization problem for which the necessary and efficient condition of a minimum is solved using the projected gradient method. Moreover, we also prove the convergence conditions and compute the estimate rate of convergence of variables corresponding to the Lagrange principle and the Tikhonov’s regularization, respectively. We provide all the details needed to implement the proposed method in an efficient way. The usefulness of the method is successfully demonstrated by a numerical example.
97 MATHEMATICS AND COMPUTING↗
Lecture notes on stochastic control, part I
Stochastic control of diffusion processes, optimal linear filtering and prediction, and optimization of control for Markov chains
CONTROL SYSTEM↗
A Multiobjective Optimization Framework for Stochastic Control of Complex Systems
This paper addresses the problem of minimizing the long-run expected average cost of a complex system consisting of subsystems that interact with each other and the environment. We treat the stochastic control problem as a multiobjective optimization problem of the one-stage expected costs of the subsystems, and we show that the control policy yielding the Pareto optimal solution is an optimal control policy that minimizes the average cost criterion for the entire system. For practical situations with constraints consistent to those we study here, our results imply that the Pareto control policy may be of value in deriving online an optimal control policy in complex systems.
Complex systems↗
Accelerated procedures for the solution of discret e Markov control problems
Accelerated procedures for Markov chain model optimal control problems solution with computation time advantage over usual dynamic programming
Kleinman, A. J.↗
Two-stage chain sampling inspection plans with different sample sizes in the two stages
Two-stage chain sampling inspection plans with different sample sizes in stages
SAMPLING↗
Stochastic Optimal Control via Bellman's Principle
This paper presents a method for finding optimal controls of nonlinear systems subject to random excitations. The method is capable to generate global control solutions when state and control constraints are present. The solution is global in the sense that controls for all initial conditions in a region of the state space are obtained. The approach is based on Bellman's Principle of optimality, the Gaussian closure and the Short-time Gaussian approximation. Examples include a system with a state-dependent diffusion term, a system in which the infinite hierarchy of moment equations cannot be analytically closed, and an impact system with a elastic boundary. The uncontrolled and controlled dynamics are studied by creating a Markov chain with a control dependent transition probability matrix via the Generalized Cell Mapping method. In this fashion, both the transient and stationary controlled responses are evaluated. The results show excellent control performances.
Crespo, Luis G.↗
Numerical methods for the solution of the degenerate non-linear elliptic equations arising in optimal stochastic control theory.
Iterative method for optimal control problems for Markov chains, discussing degenerate nonlinear elliptic equations solution in optimal stochastic control theory
Kleinman, A.↗
A Markovian reset problem
Optimal control of systems with time transitions described by finite state stationary Markov chain, noting resetting control
Kolesar, P.↗
Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory.
Optimal control problems for Markov chains solved by iterative method, using nonlinear finite difference equations to approximate degenerate elliptic functions
Kleinman, A. J.↗
Markov chains for testing redundant software
A preliminary design for a validation experiment has been developed that addresses several problems unique to assuring the extremely high quality of multiple-version programs in process-control software. The procedure uses Markov chains to model the error states of the multiple version programs. The programs are observed during simulated process-control testing, and estimates are obtained for the transition probabilities between the states of the Markov chain. The experimental Markov chain model is then expanded into a reliability model that takes into account the inertia of the system being controlled. The reliability of the multiple version software is computed from this reliability model at a given confidence level using confidence intervals obtained for the transition probabilities during the experiment. An example demonstrating the method is provided.
White, Allan L.↗
Using Markov Models of Fault Growth Physics and Environmental Stresses to Optimize Control Actions
A generalized Markov chain representation of fault dynamics is presented for the case that available modeling of fault growth physics and future environmental stresses can be represented by two independent stochastic process models. A contrived but representatively challenging example will be presented and analyzed, in which uncertainty in the modeling of fault growth physics is represented by a uniformly distributed dice throwing process, and a discrete random walk is used to represent uncertain modeling of future exogenous loading demands to be placed on the system. A finite horizon dynamic programming algorithm is used to solve for an optimal control policy over a finite time window for the case that stochastic models representing physics of failure and future environmental stresses are known, and the states of both stochastic processes are observable by implemented control routines. The fundamental limitations of optimization performed in the presence of uncertain modeling information are examined by comparing the outcomes obtained from simulations of an optimizing control policy with the outcomes that would be achievable if all modeling uncertainties were removed from the system.
Bole, Brian↗
Data-based Aggregate Model of Refrigerators for Electric Power Grid Services
Electrical power system stability under high penetration of intermittent renewable energy generation, such as wind and solar energy, has become a high concern. It is unpredictable and hard to control due to little inertial behavior characteristic to these generation resources. To help mitigate possibly undesirable frequency excursions, demand-side frequency control presents an emerging solution. By observing the frequency deviation, devices could switch operating modes for a short time to rebalance generation. To analyze device level control algorithms, aggregate models for their behavior are needed. Refrigerators are household devices that could help with load modulation. This paper uses real measured data to analyze the refrigerator behavior, learn about its operating mode and power consumption ranges. A Markov chain process is proposed to model the transition of a refrigerator population from one operating mode to another, such that the overall load is modulated according to the control input. The Markov-chain model can be used to design control so that the overall load is modulated according to the control input.
Markov chain model, demand response, refrigerator,↗
Bayesian Inversion of Seismic and Electromagnetic Data for Marine Gas Reservoir Characterization Using Multi-chain Markov Chain Monte Carlo Sampling
In this study we developed an efficient Bayesian inversion framework for interpreting marine seismic amplitude versus angle (AVA) and controlled source electromagnetic (CSEM) data for marine reservoir characterization. The framework uses a multi-chain Markov-chain Monte Carlo (MCMC) sampler, which is a hybrid of DiffeRential Evolution Adaptive Metropolis (DREAM) and Adaptive Metropolis (AM) samplers. The inversion framework is tested by estimating reservoir-fluid saturations and porosity based on marine seismic and CSEM data. The multi-chain MCMC is scalable in terms of the number of chains, and is useful for computationally demanding Bayesian model calibration in scientific and engineering problems. As a demonstration, the approach is used to efficiently and accurately estimate the porosity and saturations in a representative layered synthetic reservoir. The results indicate that the seismic AVA and CSEM joint inversion provides better estimation of reservoir saturations than the seismic AVA-only inversion, especially for the parameters in deep layers. The performance of the inversion approach for various levels of noise in observational data was evaluated – reasonable estimates can be obtained with noise levels up to 25%. Sampling efficiency due to the use of multiple chains was also checked and was found to have almost linear scalability.
Ren, Huiying↗