Search NASA⌕ Search

SEARCH · Search NASA

Results for “Policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Data and Thread Placement in NUMA Architectures: A Statistical Learning Approach

Nowadays, NUMA architectures are common in compute-intensive systems. Achieving high performance for multi-threaded application requires both a careful placement of threads on computing units and a thorough allocation of data in memory. Finding such a placement is a hard problem to solve, because performance depends on complex interactions in several layers of the memory hierarchy. In this paper we propose a black-box approach to decide if an application execution time can be impacted by the placement of its threads and data, and in such a case, to choose the best placement strategy to adopt. We show that it is possible to reach near-optimal placement policy selection. Furthermore, solutions work across several recent processor architectures and decisions can be taken with a single run of low overhead profiling.

data↗

Reinforcement learning applied to dilute combustion control for increased fuel efficiency

To reduce the modeling burden for control of spark-ignition engines, reinforcement learning (RL) has been applied to solve the dilute combustion limit problem. Q-learning was used to identify an optimal control policy to adjust the fuel injection quantity in each combustion cycle. A physics-based model was used to determine the relevant states of the system used for training the control policy in a data-efficient manner. The cost function was chosen such that high cycle-to-cycle variability (CCV) at the dilute limit was minimized while maintaining stoichiometric combustion as much as possible. Experimental results demonstrated a reduction of CCV after the training period with slightly lean combustion, contributing to a net increase in fuel conversion efficiency of 1.33%. To ensure stoichiometric combustion for three-way catalyst compatibility, a second feedback loop based on an exhaust oxygen sensor was incorporated into the fuel quantity controller using a slow proportional-integral (PI) controller. The closed-loop experiments showed that both feedback loops can cooperate effectively, maintaining stoichiometric combustion while reducing combustion CCV and increasing fuel conversion efficiency by 1.09%. Finally, a modified cost function was proposed to ensure stoichiometric combustion with a single controller. In addition, the learning period was shortened by half to evaluate the RL algorithm performance on limited training time. Experimental results showed that the modified cost function could achieve the desired CCV targets, however, the learning time was reduced by half and the fuel conversion efficiency increased only by 0.30%.

33 ADVANCED PROPULSION SYSTEMS↗

Decision-making based on Markov decision process in integrated artificial reasoning framework—Part I: Theory

This paper presents a decision-making framework based on an integrated artificial reasoning framework and Markov decision process (MDP). The integrated artificial reasoning framework provides a physics-based approach that converts system information into state transition models, and the analysis result will be represented by the transition probabilities that can be used with an MDP to find a traceable and explainable optimal pathway. A dynamic Bayesian network (DBN) is well suited for representing the structure of an MDP. The causality information among process variables (or among subsystems) is mathematically represented in a DBN by the conditional probabilities of the node’s states provided different probabilities of the parent node’s states. To define node states in a physically understandable manner, we used multilevel flow modeling (MFM). An MFM follows the fundamental energy and mass conservation laws and supports the selection of process variables that represent the system of interest so that causal relations among process variables are properly captured. An MFM-based DBN supports developing state transition models in an MDP to capture the effect of process variables of system having physical relations. The operators of the target system can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from component degradation or random failures. We analyzed a simplified exemplary system to illustrate an optimal operational policy using the suggested approach.

Markov decision process↗

Sequential Decision Making (SDM) for Mesh Refinement and Model Selection in Multiscale, Multi-Physics Applications

Intelligent automation and decision support are needed to enhance computational efficiency and robustness in multiscale and multi-physics problems, including materials science, manufacturing, and climate and weather modeling. Current scientific computing approaches for enabling decisions by scientists fail to explore the role of learning, reasoning, and probabilistic planning. Often these decisions are not performed in real-time during the computation but are made prior to the start of the computation, which must be interrupted in order to make changes to the prior choices. Such interruptions at different stages of the computation increase the total computing time and the need for a human expert to frequently monitor the results. State of art scientific computing methods consist of rule-based algorithms that cannot automatically adapt to a dynamically changing computing environment. The development of a Sequential Decision Making (SDM) framework will automate scientific computing by optimizing the policies for mesh refinement, time-stepping, model and algorithm selection, resource allocation, and pre and post-processing. Our agent SDM framework for scientific computing will consist of data-driven learning (Classifier), automated reasoning (contextual knowledge), and probabilistic planning (Reinforcement Learning). In this project, we focused on three problems to demonstrate our SDM framework on a set of ordinary and partial differential equations. Classification of Lorenz system regions using Feed-Forward Neural Networks examined learning in the SDM framework. On the other hand, reasoning and planning in the SDM framework were used in two problems: adaptive time-stepping for nonlinear ODEs using on-policy RL algorithms, and adaptive mesh refinement for 2-D PDEs using off-policy RL algorithms.

97 MATHEMATICS AND COMPUTING↗

SCORE (Synthesis of COnsists as Rolling Energy microgrids) (Final Technical Report)

The SCORE toolset is an open source, web-based application to assess the impact of new powering technologies on railroad performance, specifically technologies capable of both putting power into the rail (motive force) and taking power from the rail (regeneration). SCORE’s primary goal is generate trade studies to analyze different powering and train make-up options to minimize energy usage and greenhouse gas generation. At the core of generating these trade spaces is calculating the optimal powering policy for the train given the makeup of the train, the route, and time constraints. This paper presents details on the algorithms used in SCORE to compute this powering policies that is fast and accurate, discusses its implementation in an Energy-Longitudinal Train Dynamics (E-LTD) model, compares it to naïve approaches, and demonstrates its use across a variety of train/route pairs.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Cutting the Deployment Costs of Physics-Based MPC in Buildings by Simulation-Based Imitation Learning

It has been shown that model predictive control (MPC) is a promising solution for energy-efficient building operations. However, the deployment of MPC in a large portion of the building stock has not been possible partially because of high installation costs. Every building is unique and requires a tailored MPC solution. The best performing solutions are often based on physics-based modeling, which is, however, computationally expensive and requires dedicated software. A promising direction that tackles this problem is to train a neural network-based optimal control policy to imitate the behavior of physics-based MPC from the simulation data generated offline. The neural networks give control actions that closely approximate those produced by physics-based MPC, but with a fraction of the computational and memory requirements and without the need for licensed software. The main advantage of the proposed approach stems from simple evaluation at execution time, leading to low computational footprints and easy deployment on embedded HW platforms. In the case study, we present the energy savings potential of physics-based MPC applied to an office building in Belgium. We demonstrate how neural network approximators can be used to cut the implementation and maintenance costs of MPC deployment without compromising performance. We also critically assess the presented approach by pointing out the remaining challenges and open research questions.

Drgona, Jan↗

Stripping off the implementation complexity of physics-based model predictive control for buildings via deep learning

Over the past decade, model predictive control (MPC) has been considered as the most promising solution for intelligent building operation. Despite extensive effort, transfer of this technology into practice is hampered by the need to obtain an accurate controller model with minimum effort, the need of expert knowledge to set it up, and the need of increased computational power and dedicated software to run it. A promising direction that tackles the last two problems was proposed by approximate explicit MPC where the optimal control policies are learned from MPC data via a suitable function approximator, e.g., a deep learning (DL) model. The main advantage of the proposed approach stems from simple evaluation at execution time leading to low computational footprints and easy deployment on embedded HW platforms. We present the energy savings potential of physics-based (also called 'white-box') MPC applied to an office building in Belgium. Moreover, we demonstrate how deep learning approximators can be used to cut the implementation and maintenance costs of MPC deployment without compromising performance. We also critically assess the presented approach by pointing out the major challenges and remaining open-research questions.

Drgona, Jan↗

Operation Optimization Using Reinforcement Learning with Integrated Artificial Reasoning Framework

In large and complex systems, operational decision-making requires a systematic analysis with a vast amount of data from both process parameters and component status monitoring. In this paper, we present an integrated artificial reasoning approach for system state transition models that can help operational decision-making with explainable and traceable reasoning. The integrated artificial reasoning framework is a physics-based approach of defining the system structure in a Bayesian network, so we leveraged it in a Markov decision process (MDP) for finding optimal operational solutions. In our proposed framework, the MDP is implemented on a dynamic Bayesian network (DBN), which represents causalities in a system. The multilevel flow modeling was utilized in order to extract these causalities in a more efficient and objective manner. Since multilevel flow modeling is based on the fundamental energy and mass conservation laws, the target system is decomposed into several mass, energy, and information structures, which serve as the basis for a DBN. The MDP consists of the processes of finding a solution for the Bellman equation, which can be derived from the conditional probability equations of the constructed DBN. System operators can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from the component degradation process or random failures. We analyzed a simplified example system to illustrate finding an optimal operational policy with this approach.

Kim, Junyung↗

Risk-Informed Operations and Maintenance Decision Making Using Deep Reinforcement Learning

A challenge for operating nuclear power plants is the significant cost of operations and maintenance, at times consuming up to 66% of annual operating costs. This project aims to build a framework for a risk-informed asset-management tool that integrates inspections, repairs, spare-part inventory, supply chain, and business choices to lower overall O&M costs. Our approach uses a combination of data-driven modeling and deep reinforcement learning to create and implement optimal maintenance policies for the existing nuclear fleet, as well as new advanced reactors. The creation of an asset management tool that uses these advanced methods will give operators new capabilities to help reduce the burden of O&M spending in nuclear power plants.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor-critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection-diffusion field. As a result, we provide explanation for the policy behaviors using knowledge of the underlying physical process.

97 MATHEMATICS AND COMPUTING↗

A hybrid inventory management system respondingto regular demand and surge demand

This paper proposes a hybrid policy for a stochastic inventory system facing regular demand and surge demand. The combination of two different demand patterns can be observed in many areas, such as healthcare inventory and humanitarian supply chain management. The surge demand has a lower arrival rate but higher demand volume per arrival. The solution approach proposed in this paper incorporates the level crossing method and mixed integer programming technique to optimize the hybrid inventory policy with both regular orders and emergency orders. The level crossing method is applied to obtain the equilibrium distributions of inventory levels under a given policy. The model is further transformed into a mixed integer program to identify an optimal hybrid policy. A sensitivity analysis is conducted to investigate the impact of parameters on the optimal inventory policy and minimum cost. Numerical results clearly show the benefit of using the proposed hybrid inventory model. The model and solution approach could help healthcare providers or humanitarian logistics providers in managing their emergency supplies in responding to surge demands.

99 GENERAL AND MISCELLANEOUS↗

A Multiobjective Optimization Framework for Online Stochastic Optimal Control in Hybrid Electric Vehicles

The increasing urgency to extract additional efficiency from hybrid propulsion systems has led to the development of advanced power management control algorithms. In this paper we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain and we show that the control policy yielding the Pareto optimal solution minimizes online the long-run expected average cost per unit time criterion. The effectiveness of the proposed solution is validated through simulation and compared to the solution derived with dynamic programming using the average cost criterion. Both solutions achieved the same cumulative fuel consumption demonstrating that the online Pareto control policy is an optimal control policy.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Shifting agricultural practices to produce sustainable, low carbon intensity feedstocks for biofuel production

The carbon intensity (CI) of biofuel's well-to-pump life cycle is calculated by life cycle analysis (LCA) to account for the energy/material inputs of the feedstock production and fuel conversion stages and the associated greenhouse gas (GHG) emissions during these stages. The LCA is used by the California Air Resources Board's Low Carbon Fuel Standard (LCFS) program to calculate CI and monetary credits are issued based on the difference between a given fuel's CI and a reference fuel's CI. Through the Tier 2 certification program under which individual fuel production facilities can submit their own CIs with their facility input data, the LCFS has driven innovative technologies to biofuel conversion facilities, resulting in substantial reductions in GHG emissions as compared to the baseline gasoline or diesel. A similar approach can be taken to allow feedstock petition in the LCFS so that lower-CI feedstock can be rewarded. Here we examined the potential for various agronomic practices to improve the GHG profiles of corn ethanol by performing feedstock-level CI analysis for the Midwestern United States. Our system boundary covers GHG emissions from the cradle-to-farm-gate activities (i.e. farm input manufacturing and feedstock production), along with the potential impacts of soil organic carbon change during feedstock production. We conducted scenario-based CI analysis of ethanol, coupled with regionalized inventory data, for various farming practices to manage corn fields, and identified key parameters affecting cradle-to-farm-gate GHG emissions. The results demonstrate large spatial variations in CI of ethanol due to farm input use and land management practices. In particular, adopting conservation tillage, reducing nitrogen fertilizer use, and implementing cover crops has the potential to reduce GHG emissions per unit corn produced when compared to a baseline scenario of corn–soybean rotation. This work shows a large potential emission offset opportunity by allowing feedstock producers a path to Tier 2 petitions that reward low-CI feedstocks and further reduce biofuels' CI. The prevalence of significant acreage that has not been optimized for CI suggests that policy changes that incentivize optimization of this parameter could provide significant additionality over current trends in farm efficiency and adoption of conservation practice.

09 BIOMASS FUELS↗

Optimization of Low Level Waste Disposal Policy, Strategies and Techniques - 17201

The mission of low level radioactive waste disposal facilities is to accept and dispose of waste in compliance with all applicable laws, regulations, license requirements, and social expectations. These facilities need to provide a cost effective disposal service to the waste generators, so that the cost savings could be passed down to business, industry, and ultimately, to the general public. To fulfil this mission, waste disposal facilities need to establish and maintain review and optimization mechanisms to respond to changing circumstances that may impact the long-term safe, cost effective operation of their facilities. Maintaining existing disposal resources available as long as possible, managing increased demand for disposal space as a consequence of growing number of nuclear facilities to be decommissioned, and developing capacity for future needs and uncertainties are some of the key challenges that drive optimization of disposal facilities. While the concept of optimization of low level waste disposal facilities is not new, the framework for holistic optimization of waste disposal facilities, enabling them to provide cost effective, sustainable services to the nuclear industry, as well as maintaining optimized safety and radiation protection systems is a notable gap in the current radioactive waste management body of knowledge. A comprehensive optimization framework is required to support the implementation of safe, effective, and sustainable integrated waste management strategies. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Justice 40 Tool (J40 Tool) v1.0

The Justice 40 tool provides a quantitative framework to support decision-making around equitable energy interventions at the community level. The tool calculates the optimal portfolio of policy interventions that explicitly mitigates energy insecurity of an eligible population, by reducing its disproportionate energy burden. The place-based analysis assumes a spatial census tract-level resolution and distinguishes different sociodemographic groups within each tract. Instead of focusing on a specific technology, the underlying J40 model captures the combined effect of a set of policy interventions, currently including weatherization, rooftop solar, community solar and community wind. For each tract, the model chooses the optimal (least cost) combination of interventions to address the disproportionate burden, considering the specific population demographics and techno-economic potentials of technologies. Mathematically, this problem is formalized as an optimization model and formulated as a linear program.

Heleno, Miguel↗

A duality framework for stochastic optimal control of complex systems

In this study, we address the problem of minimizing the long-run expected average cost of a complex system consisting of interactive subsystems. We formulate a multiobjective optimization problem of the one-stage expected costs of the subsystems and provide a duality framework to prove that the control policy yielding the Pareto optimal solution minimizes the average cost criterion of the system. We provide the conditions of existence and a geometric interpretation of the solution. For practical situations having constraints consistent with those studied here, our results imply that the Pareto control policy may be of value when we seek to derive online the optimal control policy in complex systems.

97 MATHEMATICS AND COMPUTING↗

Learning to Optimize Variational Quantum Circuits to Solve Combinatorial Problems

Quantum computing is a computational paradigm with the potential to outperform classical methods for a variety of problems. Proposed recently, the Quantum Approximate Optimization Algorithm (QAOA) is considered as one of the leading candidates for demonstrating quantum advantage in the near term. QAOA is a variational hybrid quantum-classical algorithm for approximately solving combinatorial optimization problems. The quality of the solution obtained by QAOA for a given problem instance depends on the performance of the classical optimizer used to optimize the variational parameters. In this paper, we formulate the problem of finding optimal QAOA parameters as a learning task in which the knowledge gained from solving training instances can be leveraged to find high-quality solutions for unseen test instances. To this end, we develop two machine-learning-based approaches. Our first approach adopts a reinforcement learning (RL) framework to learn a policy network to optimize QAOA circuits. Our second approach adopts a kernel density estimation (KDE) technique to learn a generative model of optimal QAOA parameters. In both approaches, the training procedure is performed on small-sized problem instances that can be simulated on a classical computer; yet the learned RL policy and the generative model can be used to efficiently solve larger problems. Furthermore, extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our proposed RL- and KDE-based approaches reduce the optimality gap by factors up to 30.15 when compared with other commonly used off-the-shelf optimizers.

97 MATHEMATICS AND COMPUTING↗