Search NASA⌕ Search

SEARCH · Search NASA

Results for “sequential decision making”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Variational Information Planning for Sequential Decision Making

We consider the setting of sequential decision making where, at each stage, potential actions are evaluated based on expected reduction in posterior uncertainty, given by mutual information (MI). As MI typically lacks a closed form, we propose an approach which maintains variational approximations of, both, the posterior and MI utility. Our planning objective extends an established variational bound on MI to the setting of sequential planning. The result, variational information planning (VIP), is an efficient method for sequential decision making. We further establish convexity of the variational planning objective and, under conditional exponential family approximations, we show that the optimal MI bound arises from a relaxation of the well-known exponential family moment matching property. Here, we demonstrate VIP for sensor selection, experiment design, and active learning, where it meets or exceeds methods requiring more computation, or those specialized to the task.

Pacheco, Jason↗
DOE OSTI2023

Correlated Trajectory Uncertainty for Adaptive Sequential Decision Making

One of the great challenges with decision making tasks on real world systems is the fact that data is sparse and acquiring additional data is expensive. In these cases, it is often crucial to make a model of the environment to assist in making decisions. At the same time, limited data means that learned models are erroneous, making it just as important to equip the model with good predictive uncertainties. In the context of learning sequential decision making policies, these uncertainties can prove useful for informing which data to collect for the greatest improvement in policy performance \citep{mehta2021experimental, mehta2022exploration} or informing the policy about unsure regions of state and action space to avoid during test time \citep{yu2020mopo}. Additionally, assuming that realistic samples of the environment can be drawn, an adaptable policy can be trained that attempts to make optimal decisions for any given possible instance of the environment \citep{ghosh2022offline, chen2021offline}. In this work, we examine the so-called ``probabilistic neural network'' (PNN) model that is ubiquitous in model-based reinforcement learning (MBRL) works. We argue that while PNN models may have good marginal uncertainties, they form a distribution of non-smooth transition functions. Not only are these samples unrealistic and may hamper adaptability, but we also assert that this leads to poor uncertainty estimates when predicting multiple step trajectory estimates. To address this issue, we propose a simple sampling method that can be implemented on top of pre-existing models.We evaluate our sampling technique on a number of environments, including a realistic nuclear fusion task, and find that, not only do smooth transition function samples produce more calibrated uncertainties, but they also lead to better downstream performance for an adaptive policy.

Offline Reinforcement Learning↗
DOE OSTI2022

Sequential Decision Making (SDM) for Mesh Refinement and Model Selection in Multiscale, Multi-Physics Applications

Intelligent automation and decision support are needed to enhance computational efficiency and robustness in multiscale and multi-physics problems, including materials science, manufacturing, and climate and weather modeling. Current scientific computing approaches for enabling decisions by scientists fail to explore the role of learning, reasoning, and probabilistic planning. Often these decisions are not performed in real-time during the computation but are made prior to the start of the computation, which must be interrupted in order to make changes to the prior choices. Such interruptions at different stages of the computation increase the total computing time and the need for a human expert to frequently monitor the results. State of art scientific computing methods consist of rule-based algorithms that cannot automatically adapt to a dynamically changing computing environment. The development of a Sequential Decision Making (SDM) framework will automate scientific computing by optimizing the policies for mesh refinement, time-stepping, model and algorithm selection, resource allocation, and pre and post-processing. Our agent SDM framework for scientific computing will consist of data-driven learning (Classifier), automated reasoning (contextual knowledge), and probabilistic planning (Reinforcement Learning). In this project, we focused on three problems to demonstrate our SDM framework on a set of ordinary and partial differential equations. Classification of Lorenz system regions using Feed-Forward Neural Networks examined learning in the SDM framework. On the other hand, reasoning and planning in the SDM framework were used in two problems: adaptive time-stepping for nonlinear ODEs using on-policy RL algorithms, and adaptive mesh refinement for 2-D PDEs using off-policy RL algorithms.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1980

Sequential decision making and stochastic networks

To solve the problems inherent in working with sequential decision processes, it is proposed to (1) utilize concepts of dominance through bounding in the decision processes (DP) formalism to reduce the amount of computing required. This advocates the marrying of DP recursion and Branch-and-Bound methodology; and (2) relax the requirement of strict optimality in the search over the state space, and be content with a tolerable error.

Elmaghraby, S.↗
DOE OSTI2016

Energy-Water Nexus: Balancing the Tradeoffs between Two-Level Decision Makers

Energy-water nexus has substantially increased importance in the recent years. Synergistic approaches based on systems-analysis and mathematical models are critical for helping decision makers better understand the interrelationships and tradeoffs between energy and water. In energywater nexus management, various decision makers with different goals and preferences, which are often conflicting, are involved. These decision makers may have different controlling power over the management objectives and the decisions. They make decisions sequentially from the upper level to the lower level, challenging decision making in energy-water nexus. In order to address such planning issues, a bi-level decision model is developed, which improves upon the existing studies by integration of bi-level programming into energy-water nexus management. The developed model represents a methodological contribution to the challenge of sequential decisionmaking in energy-water nexus through provision of an integrated modeling framework/tool. An interactive fuzzy optimization methodology is introduced to seek a satisfactory solution to meet the overall satisfaction of the two-level decision makers. The tradeoffs between the two-level decision makers in energy-water nexus management are effectively addressed and quantified. Application of the proposed model to a synthetic example problem has demonstrated its applicability in practical energy-water nexus management. Optimal solutions for electricity generation, fuel supply, water supply including groundwater, surface water and recycled water, capacity expansion of the power plants, and GHG emission control are generated. In conclusion, these analyses are capable of helping decision makers or stakeholders adjust their tolerances to make informed decisions to achieve the overall satisfaction of energy-water nexus management where bi-level sequential decision making process is involved.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Scalable Bayesian optimization with randomized prior networks

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1994

Human-Computer Interaction with Medical Decisions Support Systems

Decision Support Systems (DSSs) have been available to medical diagnosticians for some time, yet their acceptance and use have not increased with advances in technology and availability of DSS tools. Medical DSSs will be necessary on future long duration space missions, because access to medical resources and personnel will be limited. Human-Computer Interaction (HCI) experts at NASA's Human Factors and Ergonomics Laboratory (HFEL) have been working toward understanding how humans use DSSs, with the goal of being able to identify and solve the problems associated with these systems. Work to date consists of identification of HCI research areas, development of a decision making model, and completion of two experiments dealing with 'anchoring'. Anchoring is a phenomenon in which the decision maker latches on to a starting point and does not make sufficient adjustments when new data are presented. HFEL personnel have replicated a well-known anchoring experiment and have investigated the effects of user level of knowledge. Future work includes further experimentation on level of knowledge, confidence in the source of information and sequential decision making.

Adolf, Jurine A.↗
DOE OSTI2018

Risk aware net load balancing in micro grids with high DER penetration

The rapid transformation of micro grids due to the accelerated integration of renewables, storage systems and IoT enabled monitoring and control has opened up new opportunities as well as challenges. Future micro grids will be characterized by high DER penetration and will require sophisticated net load balancing frameworks which explicitly consider the errors in prediction of load and generation due to uncertainty in weather conditions while making decisions. Traditional techniques for grid net load management which rely on isolated shaping of load and supply curves are inadequate and inefficient. For micro grids with high PV penetration, the intermittent and unpredictable nature of PV based energy generation can lead to dramatic and sudden supply demand imbalances thus requiring a holistic framework for balancing net load over the entire horizon. In this paper, we develop a sequential decision making framework for net load management that optimally balances the usage of storage and energy market transactions as a mechanism for mitigating supply demand imbalances (net load imbalances) over the horizon. Our framework specifically accounts for prediction uncertainty of future net load imbalances and minimizes the tail end risk of storage shortfall at the end of the horizon. Using qualitative analysis, we show that our framework achieves its objective of minimum cost net load balancing while accounting for the tail end risk.

14 SOLAR ENERGY↗
DOE OSTI2018

A Decentralized Bi-Level Fuzzy Two-Stage Decision Model for Flood Management

Flood, as a serious worldwide environment problem, can lead to detrimental economic losses and fatalities. Effective flood control is desired to mitigate the adverse impacts of flooding and the associated flood risk through development of cost-effective and efficient flood management decisions and policies. A bi-level fuzzy two-stage stochastic programming model, named BIFS model is developed in this study to provide decision support for economic analysis of flood management. The BIFS model is capable of not only addressing the sequential decision making issue involving the two-level decision makers, but also correcting the pre-regulated flood management decisions before the occurrence of a flood event in the two-stage environment. The probabilistic and non-probabilistic uncertainties expressed as probability density functions and fuzzy sets are quantitatively analyzed. The overall satisfaction solution is obtained for meeting the goals of the two-level decision makers by compromising, reflecting the tradeoffs among various decision makers in the two decision-making levels. The results of application of the BIFS model to a representative case study indicate informed decision strategies for flood management. Tradeoffs between economic objectives and uncertainty-averse attitudes of decision makers are quantified.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Reinforcement learning building control approach harnessing imitation learning

Reinforcement learning (RL) has shown significant success in sequential decision making in fields like autonomous vehicles, robotics, marketing and gaming industries. This success has attracted the attention to the RL control approach for building energy systems which are becoming complicated due to the need to optimize for multiple, potentially conflicting, goals like occupant comfort, energy use and grid interactivity. However, for real world applications, RL has several drawbacks like requiring large training data and time, and unstable control behavior during the early exploration process making it infeasible for an application directly to building control tasks. To address these issues, an imitation learning approach is utilized herein where the RL agents starts with a policy transferred from accepted rule based policies and heuristic policies. This approach is successful in reducing the training time, preventing the unstable early exploration behavior and improving upon an accepted rule-based policy - all of these make RL a more practical control approach for real world applications in the domain of building controls.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Thermal Reservoir Networks for Modularly Expandable Thermal Microgrids

The Department of Defense (DoD) faces the substantial challenge of cost-effectively retrofitting one to two installations per month, each comprising approximately 1,000 buildings, to improve resilience, reduce energy consumption, and enhance energy supply security. Achieving these objectives requires optimal system selection and effective risk mitigation during system integration. To address this need, we introduce Platform-Based Design (PBD), a structured, hierarchical methodology adapted from other industrial sectors to the domain of energy system retrofits. We demonstrate the effectiveness of PBD through a techno-economic feasibility study comparing geothermal-coupled thermal energy networks (TENs) with conventional energy systems for heating, cooling, and powering 17 buildings at Joint Base Andrews (JBA) in Maryland. Our analysis illustrates that the PBD approach enables rigorous, data-driven, sequential decision making, resulting in a family of Pareto-optimal systems, among which the TEN emerged as the most promising solution. The selected TEN design integrates geothermal borefields, heat recovery heat pumps, photovoltaic (PV) arrays, and battery storage. Compared to the baseline system – gas heating combined with air-source chillers – the proposed TEN reduces annual imported energy by 74% and peak electricity demand by 45%, achieves a levelized cost of energy of $\$0.210$/kWh, and substantially enhances resilience. Life-cycle costs increase by approximately 6%, and initial investment costs are about 2.5 times higher than the baseline. However, if central plant infrastructure, district loops, and utility-scale PV and battery systems are privately funded and operated, the initial investment would fall below the baseline system cost. Critical to achieving these significant performance improvements were detailed nonlinear dynamic simulations coupling geothermal heat transfer, energy system operation, and realistic feedback control logic. These simulations identified essential design modifications and control strategy refinements that substantially reduced energy use, peak demand, and compressor shortcycling, thereby improving durability and reliability—issues that would have been significantly more expensive to resolve during operation. Additionally, the verification step highlighted sensitivities to key design parameters that could reduce initial investment by approximately $\$2$ million and reduce annual life-cycle costs more than $\$300,000$. We recommend adopting the PBD methodology for future feasibility studies and TEN pilot projects to gain valuable operational experience. Furthermore, we recommend that DoD invest in transferring and scaling the PBD methodology to other installations. This entails developing standardized computational frameworks and component libraries as well as training industry in conducting PBD. Such investments would enable rapid, robust, reliable, and cost-effective retrofits, supporting DoD’s ambitious energy system modernization goals.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1997

Collaborative, Sequential and Isolated Decisions in Design

The Massachusetts Institute of Technology (MIT) Commission on Industrial Productivity, in their report Made in America, found that six recurring weaknesses were hampering American manufacturing industries. The two weaknesses most relevant to product development were 1) technological weakness in development and production, and 2) failures in cooperation. The remedies to these weaknesses are considered the essential twin pillars of CE: 1) improved development process, and 2) closer cooperation. In the MIT report, it is recognized that total cooperation among teams in a CE environment is rare in American industry, while the majority of the design research in mathematically modeling CE has assumed total cooperation. In this paper, we present mathematical constructs, based on game theoretic principles, to model degrees of collaboration characterized by approximate cooperation, sequential decision making and isolation. The design of a pressure vessel and a passenger aircraft are included as illustrative examples.

Lewis, Kemper↗
NASA NTRS2003

Transition-Independent Decentralized Markov Decision Processes

There has been substantial progress with formal models for sequential decision making by individual agents using the Markov decision process (MDP). However, similar treatment of multi-agent systems is lacking. A recent complexity result, showing that solving decentralized MDPs is NEXP-hard, provides a partial explanation. To overcome this complexity barrier, we identify a general class of transition-independent decentralized MDPs that is widely applicable. The class consists of independent collaborating agents that are tied up by a global reward function that depends on both of their histories. We present a novel algorithm for solving this class of problems and examine its properties. The result is the first effective technique to solve optimally a class of decentralized MDPs. This lays the foundation for further work in this area on both exact and approximate solutions.

Becker, Raphen↗
DOE OSTI2022

A Hybrid Reinforcement Learning-MPC Approach for Distribution System Critical Load Restoration: Preprint

This paper proposes a hybrid control approach for distribution system critical load restoration, combining deep reinforcement learning (RL) and model predictive control (MPC) aiming at maximizing total restored load following an extreme event. RL determines a policy for quantifying operating reserve requirements, thereby hedging against uncertainty, while MPC models grid operations incorporating RL policy actions, i.e., the reserve requirement, renewable (wind and solar) power predictions, and load demand forecasts. We formulate the reserve requirement determination problem as a sequential decision making problem based on the Markov Decision Process (MDP) and design an RL learning environment based on the OpenAI Gym framework and MPC. The RL agent reward and MPC objective function aim to maximize and monotonically increase total restored load and minimize load shedding and renewable power curtailment. The RL algorithm is trained off-line using historical forecast of renewable generation and load demand. The method is tested using a modified IEEE 13-bus distribution test feeder containing wind turbine, photovoltaic, microturbine and battery. Case studies demonstrated that the proposed method outperforms other operating reserve determination methods.

distribution system↗
DOE OSTI2023

Role of reinforcement learning for risk-based robust control of cyber-physical energy systems

Critical infrastructures such as cyber-physical energy systems (CPS-E) integrate information flow and physical operations that are vulnerable to natural and targeted failures. Safe, secure, and reliable operation and control of CPS-E is critical to ensure societal well-being and economic prosperity. Automated control is key for real-time operations and may be mathematically cast as a sequential decision-making problem under uncertainty. Emergence of data-driven techniques for decision making under uncertainty, such as reinforcement learning (RL), have led to promising advances for addressing sequential decision-making problems for risk-based robust CPS-E control. However, existing research challenges include understanding the applicability of RL methods across diverse CPS-E applications, addressing the effect of risk preferences across multiple RL methods, and development of open-source domain-aware simulation environments for RL experimentation within a CPS-E context. This article systematically analyzes the applicability of four types of RL methods (model-free, model-based, hybrid model-free and model-based, and hierarchical) for risk-based robust CPS-E control. Problem features and solution stability for the RL methods are also discussed. We demonstrate and compare the performance of multiple RL methods under different risk specifications (risk-averse, risk-neutral, and risk-seeking) through the development and application of an open-source simulation environment. Motivating numerical simulation examples include representative single-zone and multizone building control use cases. Finally, six key insights for future research and broader adoption of RL methods are identified, with specific emphasis on problem features, algorithmic explainability, and solution stability.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A control-inspired approach for energy transition planning under uncertainty

As the global carbon footprint continues to grow, many countries are implementing carbon emission reduction policies which have incentivized the expansion of low-carbon and renewable technologies. However, the speed and scale of deployment falls short of that needed to meet climate goals. Energy system models serve as key tools for guiding investment decisions and helping policymakers evaluate the effects of various policies on the development of an energy system. This study focuses on the energy system of the United States and builds upon prior work by incorporating more geographic granularity to account for the trade of commodities and addresses transmission congestion through electricity price adjustments. Furthermore, real-world characteristics, such as delays in constructing new liquid fuel production and electricity generation facilities, are integrated using a sequential decision-making approach that better reflects how decisions can be updated as uncertainties unfold. Results demonstrate that stochastic programming combined with sequential decision-making produces energy transition pathways that are robust to multiple uncertain futures. Additionally, considering real-world characteristics significantly impacts the deployment of renewable technologies and the ability to meet carbon emission reduction goals while also reliably meeting demand. These findings highlight the importance of accounting for uncertainty and real-world characteristics to avoid overly optimistic projections in energy system planning.

energy systems↗