Search NASA⌕ Search

SEARCH · Search NASA

Results for “sequential decision making”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Variational Information Planning for Sequential Decision Making

We consider the setting of sequential decision making where, at each stage, potential actions are evaluated based on expected reduction in posterior uncertainty, given by mutual information (MI). As MI typically lacks a closed form, we propose an approach which maintains variational approximations of, both, the posterior and MI utility. Our planning objective extends an established variational bound on MI to the setting of sequential planning. The result, variational information planning (VIP), is an efficient method for sequential decision making. We further establish convexity of the variational planning objective and, under conditional exponential family approximations, we show that the optimal MI bound arises from a relaxation of the well-known exponential family moment matching property. Here, we demonstrate VIP for sensor selection, experiment design, and active learning, where it meets or exceeds methods requiring more computation, or those specialized to the task.

Pacheco, Jason↗
DOE OSTI2023

Correlated Trajectory Uncertainty for Adaptive Sequential Decision Making

One of the great challenges with decision making tasks on real world systems is the fact that data is sparse and acquiring additional data is expensive. In these cases, it is often crucial to make a model of the environment to assist in making decisions. At the same time, limited data means that learned models are erroneous, making it just as important to equip the model with good predictive uncertainties. In the context of learning sequential decision making policies, these uncertainties can prove useful for informing which data to collect for the greatest improvement in policy performance \citep{mehta2021experimental, mehta2022exploration} or informing the policy about unsure regions of state and action space to avoid during test time \citep{yu2020mopo}. Additionally, assuming that realistic samples of the environment can be drawn, an adaptable policy can be trained that attempts to make optimal decisions for any given possible instance of the environment \citep{ghosh2022offline, chen2021offline}. In this work, we examine the so-called ``probabilistic neural network'' (PNN) model that is ubiquitous in model-based reinforcement learning (MBRL) works. We argue that while PNN models may have good marginal uncertainties, they form a distribution of non-smooth transition functions. Not only are these samples unrealistic and may hamper adaptability, but we also assert that this leads to poor uncertainty estimates when predicting multiple step trajectory estimates. To address this issue, we propose a simple sampling method that can be implemented on top of pre-existing models.We evaluate our sampling technique on a number of environments, including a realistic nuclear fusion task, and find that, not only do smooth transition function samples produce more calibrated uncertainties, but they also lead to better downstream performance for an adaptive policy.

Offline Reinforcement Learning↗
DOE OSTI2022

Sequential Decision Making (SDM) for Mesh Refinement and Model Selection in Multiscale, Multi-Physics Applications

Intelligent automation and decision support are needed to enhance computational efficiency and robustness in multiscale and multi-physics problems, including materials science, manufacturing, and climate and weather modeling. Current scientific computing approaches for enabling decisions by scientists fail to explore the role of learning, reasoning, and probabilistic planning. Often these decisions are not performed in real-time during the computation but are made prior to the start of the computation, which must be interrupted in order to make changes to the prior choices. Such interruptions at different stages of the computation increase the total computing time and the need for a human expert to frequently monitor the results. State of art scientific computing methods consist of rule-based algorithms that cannot automatically adapt to a dynamically changing computing environment. The development of a Sequential Decision Making (SDM) framework will automate scientific computing by optimizing the policies for mesh refinement, time-stepping, model and algorithm selection, resource allocation, and pre and post-processing. Our agent SDM framework for scientific computing will consist of data-driven learning (Classifier), automated reasoning (contextual knowledge), and probabilistic planning (Reinforcement Learning). In this project, we focused on three problems to demonstrate our SDM framework on a set of ordinary and partial differential equations. Classification of Lorenz system regions using Feed-Forward Neural Networks examined learning in the SDM framework. On the other hand, reasoning and planning in the SDM framework were used in two problems: adaptive time-stepping for nonlinear ODEs using on-policy RL algorithms, and adaptive mesh refinement for 2-D PDEs using off-policy RL algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Energy-Water Nexus: Balancing the Tradeoffs between Two-Level Decision Makers

Energy-water nexus has substantially increased importance in the recent years. Synergistic approaches based on systems-analysis and mathematical models are critical for helping decision makers better understand the interrelationships and tradeoffs between energy and water. In energywater nexus management, various decision makers with different goals and preferences, which are often conflicting, are involved. These decision makers may have different controlling power over the management objectives and the decisions. They make decisions sequentially from the upper level to the lower level, challenging decision making in energy-water nexus. In order to address such planning issues, a bi-level decision model is developed, which improves upon the existing studies by integration of bi-level programming into energy-water nexus management. The developed model represents a methodological contribution to the challenge of sequential decisionmaking in energy-water nexus through provision of an integrated modeling framework/tool. An interactive fuzzy optimization methodology is introduced to seek a satisfactory solution to meet the overall satisfaction of the two-level decision makers. The tradeoffs between the two-level decision makers in energy-water nexus management are effectively addressed and quantified. Application of the proposed model to a synthetic example problem has demonstrated its applicability in practical energy-water nexus management. Optimal solutions for electricity generation, fuel supply, water supply including groundwater, surface water and recycled water, capacity expansion of the power plants, and GHG emission control are generated. In conclusion, these analyses are capable of helping decision makers or stakeholders adjust their tolerances to make informed decisions to achieve the overall satisfaction of energy-water nexus management where bi-level sequential decision making process is involved.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Scalable Bayesian optimization with randomized prior networks

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Risk aware net load balancing in micro grids with high DER penetration

The rapid transformation of micro grids due to the accelerated integration of renewables, storage systems and IoT enabled monitoring and control has opened up new opportunities as well as challenges. Future micro grids will be characterized by high DER penetration and will require sophisticated net load balancing frameworks which explicitly consider the errors in prediction of load and generation due to uncertainty in weather conditions while making decisions. Traditional techniques for grid net load management which rely on isolated shaping of load and supply curves are inadequate and inefficient. For micro grids with high PV penetration, the intermittent and unpredictable nature of PV based energy generation can lead to dramatic and sudden supply demand imbalances thus requiring a holistic framework for balancing net load over the entire horizon. In this paper, we develop a sequential decision making framework for net load management that optimally balances the usage of storage and energy market transactions as a mechanism for mitigating supply demand imbalances (net load imbalances) over the horizon. Our framework specifically accounts for prediction uncertainty of future net load imbalances and minimizes the tail end risk of storage shortfall at the end of the horizon. Using qualitative analysis, we show that our framework achieves its objective of minimum cost net load balancing while accounting for the tail end risk.

14 SOLAR ENERGY↗
DOE OSTI2018

A Decentralized Bi-Level Fuzzy Two-Stage Decision Model for Flood Management

Flood, as a serious worldwide environment problem, can lead to detrimental economic losses and fatalities. Effective flood control is desired to mitigate the adverse impacts of flooding and the associated flood risk through development of cost-effective and efficient flood management decisions and policies. A bi-level fuzzy two-stage stochastic programming model, named BIFS model is developed in this study to provide decision support for economic analysis of flood management. The BIFS model is capable of not only addressing the sequential decision making issue involving the two-level decision makers, but also correcting the pre-regulated flood management decisions before the occurrence of a flood event in the two-stage environment. The probabilistic and non-probabilistic uncertainties expressed as probability density functions and fuzzy sets are quantitatively analyzed. The overall satisfaction solution is obtained for meeting the goals of the two-level decision makers by compromising, reflecting the tradeoffs among various decision makers in the two decision-making levels. The results of application of the BIFS model to a representative case study indicate informed decision strategies for flood management. Tradeoffs between economic objectives and uncertainty-averse attitudes of decision makers are quantified.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Reinforcement learning building control approach harnessing imitation learning

Reinforcement learning (RL) has shown significant success in sequential decision making in fields like autonomous vehicles, robotics, marketing and gaming industries. This success has attracted the attention to the RL control approach for building energy systems which are becoming complicated due to the need to optimize for multiple, potentially conflicting, goals like occupant comfort, energy use and grid interactivity. However, for real world applications, RL has several drawbacks like requiring large training data and time, and unstable control behavior during the early exploration process making it infeasible for an application directly to building control tasks. To address these issues, an imitation learning approach is utilized herein where the RL agents starts with a policy transferred from accepted rule based policies and heuristic policies. This approach is successful in reducing the training time, preventing the unstable early exploration behavior and improving upon an accepted rule-based policy - all of these make RL a more practical control approach for real world applications in the domain of building controls.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Thermal Reservoir Networks for Modularly Expandable Thermal Microgrids

The Department of Defense (DoD) faces the substantial challenge of cost-effectively retrofitting one to two installations per month, each comprising approximately 1,000 buildings, to improve resilience, reduce energy consumption, and enhance energy supply security. Achieving these objectives requires optimal system selection and effective risk mitigation during system integration. To address this need, we introduce Platform-Based Design (PBD), a structured, hierarchical methodology adapted from other industrial sectors to the domain of energy system retrofits. We demonstrate the effectiveness of PBD through a techno-economic feasibility study comparing geothermal-coupled thermal energy networks (TENs) with conventional energy systems for heating, cooling, and powering 17 buildings at Joint Base Andrews (JBA) in Maryland. Our analysis illustrates that the PBD approach enables rigorous, data-driven, sequential decision making, resulting in a family of Pareto-optimal systems, among which the TEN emerged as the most promising solution. The selected TEN design integrates geothermal borefields, heat recovery heat pumps, photovoltaic (PV) arrays, and battery storage. Compared to the baseline system – gas heating combined with air-source chillers – the proposed TEN reduces annual imported energy by 74% and peak electricity demand by 45%, achieves a levelized cost of energy of $\$0.210$/kWh, and substantially enhances resilience. Life-cycle costs increase by approximately 6%, and initial investment costs are about 2.5 times higher than the baseline. However, if central plant infrastructure, district loops, and utility-scale PV and battery systems are privately funded and operated, the initial investment would fall below the baseline system cost. Critical to achieving these significant performance improvements were detailed nonlinear dynamic simulations coupling geothermal heat transfer, energy system operation, and realistic feedback control logic. These simulations identified essential design modifications and control strategy refinements that substantially reduced energy use, peak demand, and compressor shortcycling, thereby improving durability and reliability—issues that would have been significantly more expensive to resolve during operation. Additionally, the verification step highlighted sensitivities to key design parameters that could reduce initial investment by approximately $\$2$ million and reduce annual life-cycle costs more than $\$300,000$. We recommend adopting the PBD methodology for future feasibility studies and TEN pilot projects to gain valuable operational experience. Furthermore, we recommend that DoD invest in transferring and scaling the PBD methodology to other installations. This entails developing standardized computational frameworks and component libraries as well as training industry in conducting PBD. Such investments would enable rapid, robust, reliable, and cost-effective retrofits, supporting DoD’s ambitious energy system modernization goals.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

A Hybrid Reinforcement Learning-MPC Approach for Distribution System Critical Load Restoration: Preprint

This paper proposes a hybrid control approach for distribution system critical load restoration, combining deep reinforcement learning (RL) and model predictive control (MPC) aiming at maximizing total restored load following an extreme event. RL determines a policy for quantifying operating reserve requirements, thereby hedging against uncertainty, while MPC models grid operations incorporating RL policy actions, i.e., the reserve requirement, renewable (wind and solar) power predictions, and load demand forecasts. We formulate the reserve requirement determination problem as a sequential decision making problem based on the Markov Decision Process (MDP) and design an RL learning environment based on the OpenAI Gym framework and MPC. The RL agent reward and MPC objective function aim to maximize and monotonically increase total restored load and minimize load shedding and renewable power curtailment. The RL algorithm is trained off-line using historical forecast of renewable generation and load demand. The method is tested using a modified IEEE 13-bus distribution test feeder containing wind turbine, photovoltaic, microturbine and battery. Case studies demonstrated that the proposed method outperforms other operating reserve determination methods.

distribution system↗
DOE OSTI2023

Role of reinforcement learning for risk-based robust control of cyber-physical energy systems

Critical infrastructures such as cyber-physical energy systems (CPS-E) integrate information flow and physical operations that are vulnerable to natural and targeted failures. Safe, secure, and reliable operation and control of CPS-E is critical to ensure societal well-being and economic prosperity. Automated control is key for real-time operations and may be mathematically cast as a sequential decision-making problem under uncertainty. Emergence of data-driven techniques for decision making under uncertainty, such as reinforcement learning (RL), have led to promising advances for addressing sequential decision-making problems for risk-based robust CPS-E control. However, existing research challenges include understanding the applicability of RL methods across diverse CPS-E applications, addressing the effect of risk preferences across multiple RL methods, and development of open-source domain-aware simulation environments for RL experimentation within a CPS-E context. This article systematically analyzes the applicability of four types of RL methods (model-free, model-based, hybrid model-free and model-based, and hierarchical) for risk-based robust CPS-E control. Problem features and solution stability for the RL methods are also discussed. We demonstrate and compare the performance of multiple RL methods under different risk specifications (risk-averse, risk-neutral, and risk-seeking) through the development and application of an open-source simulation environment. Motivating numerical simulation examples include representative single-zone and multizone building control use cases. Finally, six key insights for future research and broader adoption of RL methods are identified, with specific emphasis on problem features, algorithmic explainability, and solution stability.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A control-inspired approach for energy transition planning under uncertainty

As the global carbon footprint continues to grow, many countries are implementing carbon emission reduction policies which have incentivized the expansion of low-carbon and renewable technologies. However, the speed and scale of deployment falls short of that needed to meet climate goals. Energy system models serve as key tools for guiding investment decisions and helping policymakers evaluate the effects of various policies on the development of an energy system. This study focuses on the energy system of the United States and builds upon prior work by incorporating more geographic granularity to account for the trade of commodities and addresses transmission congestion through electricity price adjustments. Furthermore, real-world characteristics, such as delays in constructing new liquid fuel production and electricity generation facilities, are integrated using a sequential decision-making approach that better reflects how decisions can be updated as uncertainties unfold. Results demonstrate that stochastic programming combined with sequential decision-making produces energy transition pathways that are robust to multiple uncertain futures. Additionally, considering real-world characteristics significantly impacts the deployment of renewable technologies and the ability to meet carbon emission reduction goals while also reliably meeting demand. These findings highlight the importance of accounting for uncertainty and real-world characteristics to avoid overly optimistic projections in energy system planning.

energy systems↗
DOE OSTI2020

Use of a controlled experiment and computational models to measure the impact of sequential peer exposures on decision making

It is widely believed that one’s peers influence product adoption behaviors. This relationship has been linked to the number of signals a decision-maker receives in a social network. But it is unclear if these same principles hold when the “pattern” by which it receives these signals vary and when peer influence is directed towards choices which are not optimal. To investigate that, we manipulate social signal exposure in an online controlled experiment using a game with human participants. Each participant in the game decides among choices with differing utilities. We observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the choices, decision-makers tend to deviate from the obvious optimal decision when their peers make a similar decision which we call the influence decision, (2) when the quantity of social signals vary over time, the forwarding probability of the influence decision and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals. To better understand how these rules of peer influence could be used in modeling applications of real world diffusion and in networked environments, we use our behavioral findings to simulate spreading dynamics in real world case studies. We specifically try to see how cumulative influence plays out in the presence of user uncertainty and measure its outcome on rumor diffusion, which we model as an example of sub-optimal choice diffusion. Together, our simulation results indicate that sequential peer effects from the influence decision overcomes individual uncertainty to guide faster rumor diffusion over time. However, when the rate of diffusion is slow in the beginning, user uncertainty can have a substantial role compared to peer influence in deciding the adoption trajectory of a piece of questionable information.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Deep reinforcement learning for class imbalance fault diagnosis of equipment in nuclear power plants

In equipment fault diagnosis in nuclear power plants, there may be far more samples in one class (e.g., a health state) than in another class (e.g., a fault state). The distribution of data in each class is highly skewed. Most machine learning algorithms are suitable for balanced training datasets. When faced with imbalanced samples, these algorithms tend to provide good identification for the majority classes and bias for the minority classes. However, the misclassification of minority classes can lead to high costs. To address the above problem, this paper develops a deep reinforcement learning-based diagnosis method that models fault diagnosis as a sequential decision-making process. At each time step, the agent receives the state of the environment represented by the training samples and then takes a diagnosis action guided by a policy. If the action is correct/incorrect, the agent receives a positive/negative reward. The reward for minority classes is higher than that for majority classes. The agent’s goal is to obtain as many cumulative rewards as possible in the process, i.e., to identify the sample as correctly as possible. Six demonstration scenarios are constructed, depending on the selected fault datasets and the designed model structures. Experiments show that the proposed method achieves a higher weighted-averaged F1 score than the classical supervised learning method in most cases of class imbalance. Finally, the proposed method has potential applications in the field of class imbalance fault diagnosis of equipment in nuclear power plants.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2020

Optimal carbon storage reservoir management through deep reinforcement learning

Model-based optimization plays a central role in energy system design and management. The complexity and high-dimensionality of many process-level models, especially those used for geosystem energy exploration and utilization, often lead to formidable computational costs when the dimension of decision space is also large. This work adopts elements of recently advanced deep learning techniques to solve a sequential decision-making problem in applied geosystem management. Specifically, a deep reinforcement learning framework was formed for optimal multiperiod planning, in which a deep Q-learning network (DQN) agent was trained to maximize rewards by learning from high-dimensional inputs and from exploitation of its past experiences. To expedite computation, deep multitask learning was used to approximate high-dimensional, multistate transition functions. Both DQN and deep multitask learning are pattern based. As a demonstration, the framework was applied to optimal carbon sequestration reservoir planning using two different types of management strategies: monitoring only and brine extraction. Both strategies are designed to mitigate potential risks due to pressure buildup. Results show that the DQN agent can identify the optimal policies to maximize the reward for given risk and cost constraints. Finally, experiments also show that knowledge the agent gained from interacting with one environment is largely preserved when deploying the same agent in other similar environments.

15 GEOTHERMAL ENERGY↗
DOE OSTI2025

Energy efficiency in industrial drying: A hybrid ultrasonic system with a novel dynamic optimization framework

Drying processes are among the most energy-consuming operations in industrial and manufacturing settings, demanding strategic selection, design, and control for enhanced efficiency. Advancing drying technologies is critical for improving sustainability, lowering energy use, reducing carbon emissions, and minimizing waste. This study explores two innovative strategies aimed at transforming drying processes into sustainable, low-carbon systems by reducing energy consumption, minimizing waste, and maintaining a strong emphasis on preserving product quality. The first strategy showcases a sub-pilot scale hybrid ultrasonic-convective dryer for agrifood products. This technology, powered by electricity (process electrification), integrates non-thermal ultrasonic dehydration with convective heating and is presented as a sustainable and energy-efficient solution that enhances eco-friendly practices. The second strategy involves introducing and implementing a novel, multiobjective, mixed integer dynamic optimization technique to determine the optimal time-dependent process parameter values for the drying operation. This optimization technique yields operating conditions that are piecewise constant in time aiming to maximize the energy efficiency of the hybrid ultrasonic-convective dryer while ensuring strict adherence to product quality constraints. By adopting the hybrid ultrasonic-convective dryer, a notable 35% improvement in energy efficiency was achieved compared to conventional hot-air drying systems for drying apple slices. The proposed optimization framework further enhanced energy efficiency by nearly 14% over the most efficient process on the identical testbed, under static operating conditions. The reported enhancements have been experimentally validated. Regarding drying time (thereby improving production yield), the developed hybrid ultrasonic-convective dryer demonstrates as much as a 41% reduction in total processing time, which is further optimized by an additional 10% using our proposed optimization framework. The research outcomes have profound implications for the design and operation of drying systems, encompassing crucial aspects such as process electrification, cost-effectiveness, energy savings, time efficiency, product yield, product quality, and process automation.

Dynamic optimization↗
DOE OSTI2021

Reinforcement Learning Configuration Interaction

Selected configuration interaction (sCI) methods exploit the sparsity of the full configuration interaction (FCI) wave function, yielding significant computational savings and wave function compression without sacrificing the accuracy. Despite recent advances in sCI methods, the selection of important determinants remains an open problem. Furthermore, we explore the possibility of utilizing reinforcement learning approaches to solve the sCI problem. By mapping the configuration interaction problem onto a sequential decision-making process, the agent learns on-the-fly which determinants to include and which to ignore, yielding a compressed wave function at near-FCI accuracy. This method, which we call reinforcement-learned configuration interaction, adds another weapon to the sCI arsenal and highlights how reinforcement learning approaches can potentially help solve challenging problems in electronic structure theory.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗