Search NASA⌕ Search

SEARCH · Search NASA

Results for “policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Barrier Function-based Reinforcement Learning for Emergency Control of Power Systems

Under voltage load shedding has been considered as a standard and effective measure to recover the voltage stability of the electric power grid under emergency and severe conditions. However, this scheme usually trips a massive amount of load which can be unnecessary and harmful to customers. Recently, deep reinforcement learning (RL) has been regarded and adopted as a promising approach that can significantly reduce the amount of load shedding. However, like most existing machine learning (ML)-based control techniques, RL control usually cannot guarantee the safety of the systems under control. In this paper, we introduce a novel safe RL method for load shedding of power systems, that can enhance the safe voltage recovery of the electric power grid after experiencing faults. Unlike standard RL method, the safe RL method integrates a Barrier function into the reward function. Consequently, the optimal control policy can render the power system to avoid the safety bounds. This method is general and can be applied to other safety-critical control problems. Numerical simulations on the 39-bus IEEE benchmark is performed to demonstrate the effectiveness of the proposed safe RL emergency control, as well as its adaptive capability to faults not seen in the training.

Vu, Thanh Long↗
DOE OSTI2024

A Transfer Learning Strategy for Improving the Data Efficiency of Deep Reinforcement Learning Control in Smart Buildings

Reinforcement learning (RL) is a powerful tool that has shown promising results in many domains such as robotics and game-playing. Because RL algorithms learn optimal control policies by continuously interacting with their environments, these algorithms require a lot of data to learn, which limits their application to a wide range of domains. For this reason, there is an immense need for improving the training and data efficiency of RL. Towards addressing this research gap, this paper proposes a transfer learning (TL) approach to improve the efficiency of the RL algorithms by reducing data need and, thus, reducing training time. To demonstrate the proposed approach, a knowledge transfer from a set of buildings to another building was conducted. The results show that the proposed TL approach is a promising method that can efficiently harness the information from similar RL tasks and reduce the data needs of RL algorithms.

Amasyali, Kadir↗
DOE OSTI2019

Data and Thread Placement in NUMA Architectures: A Statistical Learning Approach

Nowadays, NUMA architectures are common in compute-intensive systems. Achieving high performance for multi-threaded application requires both a careful placement of threads on computing units and a thorough allocation of data in memory. Finding such a placement is a hard problem to solve, because performance depends on complex interactions in several layers of the memory hierarchy. In this paper we propose a black-box approach to decide if an application execution time can be impacted by the placement of its threads and data, and in such a case, to choose the best placement strategy to adopt. We show that it is possible to reach near-optimal placement policy selection. Furthermore, solutions work across several recent processor architectures and decisions can be taken with a single run of low overhead profiling.

data↗
DOE OSTI2024

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

To reduce the modeling burden for control of spark-ignition engines, reinforcement learning (RL) has been applied to solve the dilute combustion limit problem. Q-learning was used to identify an optimal control policy to adjust the fuel injection quantity in each combustion cycle. A physics-based model was used to determine the relevant states of the system used for training the control policy in a data-efficient manner. The cost function was chosen such that high cycle-to-cycle variability (CCV) at the dilute limit was minimized while maintaining stoichiometric combustion as much as possible. Experimental results demonstrated a reduction of CCV after the training period with slightly lean combustion, contributing to a net increase in fuel conversion efficiency of 1.33%. To ensure stoichiometric combustion for three-way catalyst compatibility, a second feedback loop based on an exhaust oxygen sensor was incorporated into the fuel quantity controller using a slow proportional-integral (PI) controller. The closed-loop experiments showed that both feedback loops can cooperate effectively, maintaining stoichiometric combustion while reducing combustion CCV and increasing fuel conversion efficiency by 1.09%. Finally, a modified cost function was proposed to ensure stoichiometric combustion with a single controller. In addition, the learning period was shortened by half to evaluate the RL algorithm performance on limited training time. Experimental results showed that the modified cost function could achieve the desired CCV targets, however, the learning time was reduced by half and the fuel conversion efficiency increased only by 0.30%.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Decision-making based on Markov decision process in integrated artificial reasoning framework—Part I: Theory

This paper presents a decision-making framework based on an integrated artificial reasoning framework and Markov decision process (MDP). The integrated artificial reasoning framework provides a physics-based approach that converts system information into state transition models, and the analysis result will be represented by the transition probabilities that can be used with an MDP to find a traceable and explainable optimal pathway. A dynamic Bayesian network (DBN) is well suited for representing the structure of an MDP. The causality information among process variables (or among subsystems) is mathematically represented in a DBN by the conditional probabilities of the node’s states provided different probabilities of the parent node’s states. To define node states in a physically understandable manner, we used multilevel flow modeling (MFM). An MFM follows the fundamental energy and mass conservation laws and supports the selection of process variables that represent the system of interest so that causal relations among process variables are properly captured. An MFM-based DBN supports developing state transition models in an MDP to capture the effect of process variables of system having physical relations. The operators of the target system can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from component degradation or random failures. We analyzed a simplified exemplary system to illustrate an optimal operational policy using the suggested approach.

Markov decision process↗
DOE OSTI2022

Sequential Decision Making (SDM) for Mesh Refinement and Model Selection in Multiscale, Multi-Physics Applications

Intelligent automation and decision support are needed to enhance computational efficiency and robustness in multiscale and multi-physics problems, including materials science, manufacturing, and climate and weather modeling. Current scientific computing approaches for enabling decisions by scientists fail to explore the role of learning, reasoning, and probabilistic planning. Often these decisions are not performed in real-time during the computation but are made prior to the start of the computation, which must be interrupted in order to make changes to the prior choices. Such interruptions at different stages of the computation increase the total computing time and the need for a human expert to frequently monitor the results. State of art scientific computing methods consist of rule-based algorithms that cannot automatically adapt to a dynamically changing computing environment. The development of a Sequential Decision Making (SDM) framework will automate scientific computing by optimizing the policies for mesh refinement, time-stepping, model and algorithm selection, resource allocation, and pre and post-processing. Our agent SDM framework for scientific computing will consist of data-driven learning (Classifier), automated reasoning (contextual knowledge), and probabilistic planning (Reinforcement Learning). In this project, we focused on three problems to demonstrate our SDM framework on a set of ordinary and partial differential equations. Classification of Lorenz system regions using Feed-Forward Neural Networks examined learning in the SDM framework. On the other hand, reasoning and planning in the SDM framework were used in two problems: adaptive time-stepping for nonlinear ODEs using on-policy RL algorithms, and adaptive mesh refinement for 2-D PDEs using off-policy RL algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

SCORE (Synthesis of COnsists as Rolling Energy microgrids) (Final Technical Report)

The SCORE toolset is an open source, web-based application to assess the impact of new powering technologies on railroad performance, specifically technologies capable of both putting power into the rail (motive force) and taking power from the rail (regeneration). SCORE’s primary goal is generate trade studies to analyze different powering and train make-up options to minimize energy usage and greenhouse gas generation. At the core of generating these trade spaces is calculating the optimal powering policy for the train given the makeup of the train, the route, and time constraints. This paper presents details on the algorithms used in SCORE to compute this powering policies that is fast and accurate, discusses its implementation in an Energy-Longitudinal Train Dynamics (E-LTD) model, compares it to naïve approaches, and demonstrates its use across a variety of train/route pairs.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1982

A Markov chain model for reliability growth and decay

A mathematical model is developed to describe a complex system undergoing a sequence of trials in which there is interaction between the internal states of the system and the outcomes of the trials. For example, the model might describe a system undergoing testing that is redesigned after each failure. The basic assumptions for the model are that the state of the system after a trial depends probabilistically only on the state before the trial and on the outcome of the trial and that the outcome of a trial depends probabilistically only on the state of the system before the trial. It is shown that under these basic assumptions, the successive states form a Markov chain and the successive states and outcomes jointly form a Markov chain. General results are obtained for the transition probabilities, steady-state distributions, etc. A special case studied in detail describes a system that has two possible state ('repaired' and 'unrepaired') undergoing trials that have three possible outcomes ('inherent failure', 'assignable-cause' 'failure' and 'success'). For this model, the reliability function is computed explicitly and an optimal repair policy is obtained.

Siegrist, K.↗
NASA NTRS1987

Control-System Design Program

Control-theory design package, Optimal Regulator Algorithms for Control of Linear Systems (ORACLS), developed to aid in design of controllers and optimal filters for systems modeled by linear, time-invariant differential and difference equations. Optimal linear quadratic regulator theory, Linear-Quadratic-Gaussian (LQG) problem, most widely accepted method of determining optimal control policy. Provides for solution to time-in-variant continuous or discrete LQG problems. Attractive to control-system designer providing rigorous tool for dealing with multi-input and multi-output dynamic systems in continuous and discrete form. CDO version written in FORTRAN IV. VAX version written in FORTRAN 77.

Frisch, Harold P.↗
NASA NTRS1988

The Feasibility of the Disturbance Accommodating Controller for Precision Antenna Pointing

The objective of this study is to investigate the feasibility of a pointing (position loop) controller for the NASA-JPL Deep Space Network (DSN) antennas using the Disturbance Accommodating Control (DAC) theory. A model that includes state dependent disturbances was developed, and an example demonstrating the noise estimator is presented as an initial phase in the controller design. The goal is to improve pointing accuracy by the removal of the systematic errors caused by the antenna misalignment as well as sensor noise and random wind and thermal disturbances. Preliminary simulation results show that the DAC technique is successful in both cancelling the imposed errors and maintaining an optimal control policy.

Gresham, L. L.↗
NASA NTRS1989

Analysis of a turning point problem in flight trajectory optimization

The optimal control policy for the aeroglide portion of the minimum fuel, orbital plane change problem for maneuvering entry vehicles is reduced to the solution of a turning point problem for the bank angle control. For this problem a turning point occurs at the minimum altitude of the flight, when the flight path angle equals zero. The turning point separates the bank angle control into two outer solutions that are valid away from the turning point. In a neighborhood of the turning point, where the bank angle changes rapidly, an inner solution is developed and matched with the two outer solutions. An asymptotic analysis of the turning point problem is given, and an analytic example is provided to illustrate the construction of the bank angle control.

Gracey, C.↗
NASA NTRS1994

Damage-Mitigating Control of Space Propulsion Systems for High Performance and Extended Life

A major goal in the control of complex mechanical system such as spacecraft rocket engine's advanced aircraft, and power plants is to achieve high performance with increased reliability, component durability, and maintainability. The current practice of decision and control systems synthesis focuses on improving performance and diagnostic capabilities under constraints that often do not adequately represent the materials degradation. In view of the high performance requirements of the system and availability of improved materials, the lack of appropriate knowledge about the properties of these materials will lead to either less than achievable performance due to overly conservative design, or over-straining of the structure leading to unexpected failures and drastic reduction of the service life. The key idea in this report is that a significant improvement in service life could be achieved by a small reduction in the system dynamic performance. The major task is to characterize the damage generation process, and then utilize this information in a mathematical form to synthesize a control law that would meet the system requirements and simultaneously satisfy the constraints that are imposed by the material and structural properties of the critical components. The concept of damage mitigation is introduced for control of mechanical systems to achieve high performance with a prolonged life span. A model of fatigue damage dynamics is formulated in the continuous-time setting, instead of a cycle-based representation, for direct application to control systems synthesis. An optimal control policy is then formulated via nonlinear programming under specified constraints of the damage rate and accumulated damage. The results of simulation experiments for the transient upthrust of a bipropellant rocket engine are presented to demonstrate efficacy of the damage-mitigating control concept.

Ray, Asok↗
NASA NTRS1995

Damage-mitigating control of a reusable rocket engine for high performance and extended life

The goal of damage mitigating control in reusable rocket engines is to achieve high performance with increased durability of mechanical structures such that functional lives of the critical components are increased. The major benefit is an increase in structural durability with no significant loss of performance. This report investigates the feasibility of damage mitigating control of reusable rocket engines. Phenomenological models of creep and thermo-mechanical fatigue damage have been formulated in the state-variable setting such that these models can be combined with the plant model of a reusable rocket engine, such as the Space Shuttle Main Engine (SSME), for synthesizing an optimal control policy. Specifically, a creep damage model of the main thrust chamber wall is analytically derived based on the theories of sandwich beam and viscoplasticity. This model characterizes progressive bulging-out and incremental thinning of the coolant channel ligament leading to its eventual failure by tensile rupture. The objective is to generate a closed form solution of the wall thin-out phenomenon in real time where the ligament geometry is continuously updated to account for the resulting deformation. The results are in agreement with those obtained from the finite element analyses and experimental observation for both Oxygen Free High Conductivity (OFHC) copper and a copper-zerconium-silver alloy called NARloy-Z. Due to its computational efficiency, this damage model is suitable for on-line applications of life prediction and damage mitigating control, and also permits parametric studies for off-line synthesis of damage mitigating control systems. The results are presented to demonstrate the potential of life extension of reusable rocket engines via damage mitigating control. The control system has also been simulated on a testbed to observe how the damage at different critical points can be traded off without any significant loss of engine performance. The research work reported here is built upon concepts derived from the disciplines of Controls, Thermo-fluids, Structures, and Materials. The concept of damage mitigation, as presented in this report, is not restricted to control of rocket engines. It can be applied to any system where structural durability is an important issue.

Ray, Asok↗
NASA NTRS1995

File-System Workload on a Scientific Multiprocessor

Many scientific applications have intense computational and I/O requirements. Although multiprocessors have permitted astounding increases in computational performance, the formidable I/O needs of these applications cannot be met by current multiprocessors a their I/O subsystems. To prevent I/O subsystems from forever bottlenecking multiprocessors and limiting the range of feasible applications, new I/O subsystems must be designed. The successful design of computer systems (both hardware and software) depends on a thorough understanding of their intended use. A system designer optimizes the policies and mechanisms for the cases expected to most common in the user's workload. In the case of multiprocessor file systems, however, designers have been forced to build file systems based only on speculation about how they would be used, extrapolating from file-system characterizations of general-purpose workloads on uniprocessor and distributed systems or scientific workloads on vector supercomputers (see sidebar on related work). To help these system designers, in June 1993 we began the Charisma Project, so named because the project sought to characterize 1/0 in scientific multiprocessor applications from a variety of production parallel computing platforms and sites. The Charisma project is unique in recording individual read and write requests-in live, multiprogramming, parallel workloads (rather than from selected or nonparallel applications). In this article, we present the first results from the project: a characterization of the file-system workload an iPSC/860 multiprocessor running production, parallel scientific applications at NASA's Ames Research Center.

Kotz, David↗
DOE OSTI2020

Cutting the Deployment Costs of Physics-Based MPC in Buildings by Simulation-Based Imitation Learning

It has been shown that model predictive control (MPC) is a promising solution for energy-efficient building operations. However, the deployment of MPC in a large portion of the building stock has not been possible partially because of high installation costs. Every building is unique and requires a tailored MPC solution. The best performing solutions are often based on physics-based modeling, which is, however, computationally expensive and requires dedicated software. A promising direction that tackles this problem is to train a neural network-based optimal control policy to imitate the behavior of physics-based MPC from the simulation data generated offline. The neural networks give control actions that closely approximate those produced by physics-based MPC, but with a fraction of the computational and memory requirements and without the need for licensed software. The main advantage of the proposed approach stems from simple evaluation at execution time, leading to low computational footprints and easy deployment on embedded HW platforms. In the case study, we present the energy savings potential of physics-based MPC applied to an office building in Belgium. We demonstrate how neural network approximators can be used to cut the implementation and maintenance costs of MPC deployment without compromising performance. We also critically assess the presented approach by pointing out the remaining challenges and open research questions.

Drgona, Jan↗
DOE OSTI2019

Stripping off the implementation complexity of physics-based model predictive control for buildings via deep learning

Over the past decade, model predictive control (MPC) has been considered as the most promising solution for intelligent building operation. Despite extensive effort, transfer of this technology into practice is hampered by the need to obtain an accurate controller model with minimum effort, the need of expert knowledge to set it up, and the need of increased computational power and dedicated software to run it. A promising direction that tackles the last two problems was proposed by approximate explicit MPC where the optimal control policies are learned from MPC data via a suitable function approximator, e.g., a deep learning (DL) model. The main advantage of the proposed approach stems from simple evaluation at execution time leading to low computational footprints and easy deployment on embedded HW platforms. We present the energy savings potential of physics-based (also called 'white-box') MPC applied to an office building in Belgium. Moreover, we demonstrate how deep learning approximators can be used to cut the implementation and maintenance costs of MPC deployment without compromising performance. We also critically assess the presented approach by pointing out the major challenges and remaining open-research questions.

Drgona, Jan↗
DOE OSTI2023

Operation Optimization Using Reinforcement Learning with Integrated Artificial Reasoning Framework

In large and complex systems, operational decision-making requires a systematic analysis with a vast amount of data from both process parameters and component status monitoring. In this paper, we present an integrated artificial reasoning approach for system state transition models that can help operational decision-making with explainable and traceable reasoning. The integrated artificial reasoning framework is a physics-based approach of defining the system structure in a Bayesian network, so we leveraged it in a Markov decision process (MDP) for finding optimal operational solutions. In our proposed framework, the MDP is implemented on a dynamic Bayesian network (DBN), which represents causalities in a system. The multilevel flow modeling was utilized in order to extract these causalities in a more efficient and objective manner. Since multilevel flow modeling is based on the fundamental energy and mass conservation laws, the target system is decomposed into several mass, energy, and information structures, which serve as the basis for a DBN. The MDP consists of the processes of finding a solution for the Bellman equation, which can be derived from the conditional probability equations of the constructed DBN. System operators can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from the component degradation process or random failures. We analyzed a simplified example system to illustrate finding an optimal operational policy with this approach.

Kim, Junyung↗