Defender Policy Evaluation and Resource Allocation With MITRE ATT&CK Data and Evaluations.
Abstract not provided.
97 MATHEMATICS AND COMPUTING↗
SEARCH · Search NASA
Results for “policy optimization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Operation Optimization using Reinforcement Learning with Integrated Artificial Reasoning Framework
In large and complex systems, operational decision-making requires a systematic analysis with a vast amount of data from both process parameters and component status monitoring. In this paper, we present an integrated artificial reasoning approach for system state transition models that can help operational decision-making with explainable and traceable reasoning. The integrated artificial reasoning framework is a physics-based approach of defining the system structure in a Bayesian network, so we leveraged it in a Markov decision process (MDP) for finding optimal operational solutions. In our proposed framework, the MDP is implemented on a dynamic Bayesian network (DBN), which represents causalities in a system. The multilevel flow modeling was utilized in order to extract these causalities in a more efficient and objective manner. Since multilevel flow modeling is based on the fundamental energy and mass conservation laws, the target system is decomposed into several mass, energy, and information structures, which serve as the basis for a DBN. The MDP consists of the processes of finding a solution for the Bellman equation, which can be derived from the conditional probability equations of the constructed DBN. System operators can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from the component degradation process or random failures. We analyzed a simplified example system to illustrate finding an optimal operational policy with this approach.
99 GENERAL AND MISCELLANEOUS↗
Implications of stop-and-go traffic on training learning-based car-following control
Learning-based car-following control (LCC) of connected and autonomous vehicles (CAVs) is gaining significant attention with the advancement of computing power and data accessibility. While the flexibility and large model capacity of model-free architecture enable LCC to potentially outperform the model-based car-following (CF) model in improving traffic efficiency and mitigating congestion, the generalizability of LCC for traffic conditions different from the training environment/dataset is not well-understood. Herein, this study seeks to explore the impact of stop-and-go traffic in the training dataset on the generalizability of LCC. It uses the characteristics of lead vehicle trajectories to describe stop-and-go traffic, and links the theory of identifiability (i.e., obtaining a unique parameter estimation result using sensor measurements) to the generalizability of behavior cloning (BC) and policy-based deep reinforcement learning (DRL). Correspondingly, the study shows theoretically that: (i) stop-and-go traffic can enable the property of identifiability and enhance the control performance of BC-based LCC in different traffic conditions; (ii) stop-and-go traffic is not necessary for DRL-based LCC to generalize to different traffic conditions; (iii) DRL-based LCC trained with only constant-speed lead vehicle trajectories (not sufficient to ensure identifiability) can be generalized to different traffic conditions; and (iv) stop-and-go traffic increases variance in the training dataset, which improves the convergence of parameter estimation while negatively impacting the convergence of DRL to the optimal control policy. Numerical experiments validate the above findings, illustrating that BC-based LCC entails comprehensive training datasets for generalizing to different traffic conditions, while DRL-based LCC can achieve generalization with simple free-flow traffic training environments. This further suggests DRL as a more promising and cost-effective LCC approach to reduce operational costs, mitigate traffic congestion, and enhance safety and mobility, which can accelerate the deployment and acceptance of CAVs.
33 ADVANCED PROPULSION SYSTEMS↗
Stochastic Learning Approach for Binary Optimization: Application to Bayesian Optimal Design of Experiments
Here, we present a novel stochastic approach to binary optimization suited for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The probabilistic objective is then solved by using a stochastic optimization routine to find an optimal observational policy. This formulation (a) is generally applicable to binary optimization problems with soft constraints and is ideal for OED and sensor placement problems; (b) does not require differentiability of the original objective function (e.g., a utility function in OED applications) with respect to the design variable, and thus it enables direct employment of sparsity-enforcing penalty functions such as $\ell_0$, without needing to utilize a continuation procedure or apply a rounding technique; (c) exhibits much lower computational cost than traditional gradient-based relaxation approaches; and (d) can be applied to both linear and nonlinear OED problems with proper choice of the utility function. The proposed approach is analyzed from an optimization perspective with detailed convergence analysis of the optimization approach and is also analyzed from a machine learning perspective with correspondence to policy gradient reinforcement learning. The approach is demonstrated numerically by using an idealized two-dimensional Bayesian linear inverse problem and validated by extensive numerical experiments carried out for sensor placement in a parameter identification setup.
97 MATHEMATICS AND COMPUTING↗
Co-Design of Transmission & Distribution for Improved Power System Planning and Operation
Work on the NYSERDA-MDAO project, presented at the OpenMDAO workshop.
capacity expansion↗
Online Learning and Pricing for Network Revenue Management with Reusable Resources
We consider a price-based network revenue management problem with multiple products and multiple reusable resources. Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources (servers). We adopt an incomplete information setting where the firm does not know the price-demand function for each product and the goal is to dynamically set prices of all products to maximize the total expected revenue of serving customers. We propose novel batched bandit learning algorithms for finding near-optimal pricing policies, and show that they admit a near-optimal cumulative regret bound of $O(J\sqrt{XT})$, where J, X, and T, are the numbers of products, candidate prices, and service periods, respectively. As part of our regret analysis, we develop the first finite-time mixing time analysis of an open network queueing system (i.e., the celebrated Jackson Network), which could be of independent interest. Our numerical studies show that the proposed approaches perform consistently well.
42 ENGINEERING↗
Economic dispatch for electricity merchant with energy storage and wind plant: State of charge based decision making considering market impact and uncertainties
Here this paper investigates how the market impact of electricity merchants and uncertainty of wind generation affect their co-optimized scheduling policy, specifically for merchants who have both energy storage and wind plants. In the existing literature, merchants' trading actions are usually assumed not to affect market prices; however, a large-scale energy storage merchant's actions can affect market prices. To this end, we approximate the electricity price by a linear function of the quantity of power traded by the merchant in the reward function to achieve decision-making incorporating the market impact. This paper utilizes the dynamic programming approach to analyze merchants' optimal multi-period decision-making incorporating market impact, uncertain wind generation, and energy storage constraints. First, our results demonstrate that for a merchant with co-located energy storage facilities and wind power plants, the energy storage's feasible state of charge (SOC) range can be segmented into four possible sub-ranges by three analytically developed SOC reference points. The unique optimal trading decision can be achieved by comparing the current energy inventory and the SOC references of the next period. Second, our results show that market impact and uncertainties substantially change the optimal storage scheduling policy by impacting the values of the reference points. To mitigate the negative effect of the merchant's market impact on buying and selling actions, the merchant may reduce the amount of generating or pumping electricity each period to maximize profit. Moreover, we identify and investigate the trade-off between market price and transaction quantity. Our findings provide co-optimized scheduling guidance for electricity merchants with co-located energy storage and renewable power plants systems.
17 WIND ENERGY↗
A control policy for systems with momentum exchange controllers
Spacecraft pointing control system with momentum exchange controllers, considering near optimal control policy for control moment gyro system
Skelton, R. E.↗
Multi-agent voltage control in distribution systems using GAN-DRL-based approach
Active distribution grids can experience voltage fluctuations and violations due to the high penetration of variable distributed energy resources (DERs). These problems might occur because of the uncertain and variable generation natures of these resources, especially solar photovoltaic resources, during panel shadowing scenarios. Volt-VAR control (VVC) is an efficient method that controls the reactive power set-points of the inverters to regulate the voltage of distribution grids. Although several VVC approaches have been proposed recently, the performance of these approaches degrades significantly if behind-the-meter solar generation data are unobservable/missing. Therefore, it is necessary to impute missing/unobservable PV data accurately to be utilized in VVC approaches. Further, this paper proposes a model-free, data-driven, centrally trained, and decentrally executed multi-agent deep reinforcement learning-based VVC architecture to regulate the voltage of distribution networks. A generative adversarial network (GAN) is incorporated to impute the unobservable PV data accurately, which improves the performance of the proposed control architecture. The proposed multi-agent-soft-actor–critic algorithm (MASAC)-based VVC technique utilizes the actual PV dataset as well as the imputed dataset from the GAN framework to learn the optimal coordinated control policy for controlling the optimal reactive power set-points of PV inverters. The effectiveness of the proposed approach is analyzed on a modified IEEE 34-bus test case with added PV inverters. The results are compared and analyzed with a base case model with no VVC and VVC with a local droop control approach, genetic algorithm optimization, and a centralized soft actor–critic-based approach. Moreover, the performance of the proposed approach is compared with that of a multi-agent VVC framework without using the PV generation data and load information as the system state. The results illustrate that the proposed method with more state input improves the voltage profile and reduces the power loss of the network across various loading and PV generation scenarios.
14 SOLAR ENERGY↗
Dynamic Transmission Line Switching Amid Wildfire-Prone Weather Under Decision-Dependent Uncertainty
During dry and windy seasons, environmental conditions significantly increase the risk of wildfires, exposing power grids to disruptions caused by transmission line failures. Wildfire propagation exacerbates grid vulnerability, potentially leading to prolonged power outages. To address this challenge, we propose a multistage optimization model that dynamically adjusts transmission grid topology in response to wildfire propagation, aiming to develop an optimal response policy. By accounting for decision-dependent uncertainty, where line survival probabilities depend on usage, we employ distributionally robust optimization to model uncertainty in line survival distributions. We adapt the stochastic nested decomposition algorithm and derive a deterministic upper bound for its finite convergence. To enhance computational efficiency, we exploit the Lagrangian dual problem structure for a faster generation of Lagrangian cuts. Using realistic data from the California transmission grid, we demonstrate the superior performance of dynamic response policies against two-stage alternatives through a comprehensive case study. In addition, after solving the multistage formulation, we construct easy-to-implement policies that significantly reduce computational burden while maintaining good performance in real-time deployment. History: Accepted by Russell Bent, Area Editor for Network Optimization: Algorithms and Applications. Funding: This work was supported by the U.S. Department of Energy, Office of Electricity [Grant DE-AC02-05CH11231]. The work of R. Jiang was supported in part by the U.S. National Science Foundation, Division of Electrical, Communications and Cyber Systems [Grant ECCS-1845980] and the U.S. Air Force Office of Scientific Research [Grant FA9550-23-1-0323]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2025.1210 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2025.1210 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .
Estrada-Garcia, Juan-Alberto↗
Proof of quasi-adaptivity for the m-measurement feedback class of stochastic control policies
Bounds on expected performance are established which show that the m-measurement feedback (mM) policy for nonlinear stochastic control performs as well or better than the open-loop optimal control policy, and thus is quasi-adaptive in the sense of Witenhausen (1966). The chain of performance inequalities indicate a tendency for the mM policy performance to improve with increasing m. It is suggested that the present analytical method, based on the construction of artificial control sequences denoted as utility controls, can be used to establish performance bounds on other well-known policies, avoiding the extensive Monte Carlo simulations necessary in comparing stochastic control policies.
Bayard, David S.↗
Unsupervised Resource Allocation with Graph Neural Networks
We present an approach for maximizing a global utility function by learning how to allocate resources in an unsupervised way. We expect interactions between allocation targets to be important and therefore propose to learn the reward structure for near-optimal allocation policies with a GNN. By relaxing the resource constraint, we can employ gradient-based optimization in contrast to more standard evolutionary algorithms. Our algorithm is motivated by a problem in modern astronomy, where one needs to select-based on limited initial information-among $10^9$ galaxies those whose detailed measurement will lead to optimal inference of the composition of the universe. Our technique presents a way of flexibly learning an allocation strategy by only requiring forward simulators for the physics of interest and the measurement process. We anticipate that our technique will also find applications in a range of resource allocation problems.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
From Optimization to Sampling Through Gradient Flows
Optimization and sampling algorithms play a central role in science and engineering as they enable finding optimal predictions, policies, and recommendations, as well as expected and equilibrium states of complex systems. The notion of “optimality” is formalized by the choice of an objective function, while the notion of an “expected” state is specified by a probabilistic model for the distribution of states. Optimizing rugged objective functions and sampling multimodal distributions is computationally challenging, especially in high-dimensional problems. Here, for this reason, many optimization and sampling methods have been developed by researchers working in disparate fields such as Bayesian statistics, molecular dynamics, genetics, quantum chemistry, machine learning, weather forecasting, econometrics, and medical imaging.
Trillos, N. García↗
Potential State Regulatory Pathways to Facilitate Low-Carbon Fuels
States and the federal government are increasingly engaged in the challenges around decarbonizing the electric grid. In particular, regulators, consumers, stakeholders, and utilities recognize the need to carefully consider the role natural gas will play in a decarbonized future. A variety of technology and policy options to reduce greenhouse gas emissions associated with natural gas use are available, including energy efficiency programs, demand reduction tools, strategic electrification, and strategies to reduce emissions from natural gas production, transportation, and consumption. Low-carbon fuels – mainly renewable natural gas (RNG) and clean hydrogen – are being considered an important component of decarbonization goals. RNG and hydrogen may be able to meaningfully reduce emissions from processes independent of geologic natural gas, displacing emissions of methane, a powerful greenhouse gas. Although RNG and hydrogen are not cost-competitive today with geologic natural gas and are smaller in scale and potential than other decarbonization options, they can be explored as potential critical tools to decarbonize sectors that are difficult to electrify or shift off of natural gas entirely, such as air travel, industrial processes, maritime transport, long-distance trucking, space heating on cold days, and railroads (Nadel, 2022). The role of this report is to provide informational context for state utility regulators to understand the impacts of and challenges associated with broader integration of low-carbon fuels, followed by examples of state regulatory actions taken to date to facilitate the development of low-carbon fuels. Setting clear guidance to calculate the environmental benefits of low-carbon fuels and continuing federal and state investments in research and development to reduce costs relative to fossil fuels will be important steps to take to signal the desire to grow the market for these fuels. State public utility commissions may play a key role in setting regulatory frameworks for low-carbon fuels and ensuring that ratepayer funds, if utilized, are done so to further the public interest. This report is intended to summarize decisions that states have made to date on low-carbon fuels. In the spirit of understanding the current market and sharing information, this report provides success stories, and lessons learned across states as regulators implement varying strategies to achieve decarbonization objectives while maintaining their focus on affordability, safety, and reliability of the energy system. The report begins with an introduction of the role of natural gas in the U.S. economy (Section I) and background information on natural gas use, decarbonization, and low-carbon fuels (Section II). Next, the report describes the current market by discussing the scale of current production, emissions intensity, resource potential, and costs of low-carbon fuels compared to geologic natural gas (Section III). Following these sections, the report describes four strategies states have employed to facilitate low-carbon fuels: opening exploratory dockets, approving voluntary tariffs for customers, approving interconnection tariffs for producers, and considering portfolio-wide procurement targets (Section IV). This section lists states that have taken actions in each category, citing utility filings, commission decisions, stakeholder comments, and other relevant sources. Finally, the report concludes with suggested questions regulators may wish to consider regarding low-carbon fuels, in the interest of preparing to make decisions in the future (Section V). These questions include: Are there existing regulatory or technical barriers to voluntary purchases of low-carbon fuels? Can customers work with utilities to procure low-carbon fuels; are producers able to interconnect projects without significant barriers to entry? Should the infrastructure and/or commodity costs of low-carbon fuels be socialized among all ratepayers, or borne solely by the large commercial and industrial (C&I) customers currently driving the market? Should regulated natural gas and/or electric utilities own and operate low-carbon fuel production? How should regulators consider the unique decarbonization potential of low-carbon fuels, particularly for hard-to-abate sectors, in decision-making? Is additional direction or clarity from state policymakers needed? What no-regrets approaches can help facilitate both near-term RNG development and long-term development of hydrogen and other zero-carbon fuels? We collectively wish to express our gratitude to the U.S. Department of Energy, Office of Fossil Energy and Carbon Management, for supporting this report and other technical assistance resources for state regulators on natural gas topics. State regulators operate under a variety of policy environments, and states have vastly different types of energy resources, infrastructure, and customers. While there is no optimal regulatory, policy, or technological solution that will be successful in every state, state regulators can benefit by exchanging lessons learned with their peers across the country. We look forward to continued engagement with our fellow commissioners, commission staff, NARUC, the U.S. Department of Energy, and other stakeholders to develop sound regulation in the public interest.
03 NATURAL GAS↗
DTS: Building custom, intelligent schedulers
DTS is a decision-theoretic scheduler, built on top of a flexible toolkit -- this paper focuses on how the toolkit might be reused in future NASA mission schedulers. The toolkit includes a user-customizable scheduling interface, and a 'Just-For-You' optimization engine. The customizable interface is built on two metaphors: objects and dynamic graphs. Objects help to structure problem specifications and related data, while dynamic graphs simplify the specification of graphical schedule editors (such as Gantt charts). The interface can be used with any 'back-end' scheduler, through dynamically-loaded code, interprocess communication, or a shared database. The 'Just-For-You' optimization engine includes user-specific utility functions, automatically compiled heuristic evaluations, and a postprocessing facility for enforcing scheduling policies. The optimization engine is based on BPS, the Bayesian Problem-Solver (1,2), which introduced a similar approach to solving single-agent and adversarial graph search problems.
Hansson, Othar↗
Incorporating Operational Uncertainties into the Dispatch of an Integrated Solar and Storage System
The economic assessment of hybrid energy systems (HES) pairing battery energy storage systems (BESSs) and photovoltaics (PV) is highly important for advancing their deployment in power systems. This paper presents an innovative assessment framework, including an optimal control policy for dispatch under uncertainty and procedures for exploring control parameters that maximize economic benefits. The proposed dispatch policy consists of two steps using system forecast information. The first step is to determine whether a BESS will be used within an operational scheduling time frame based on the probability of events and their thresholds. Once the dispatch of BESS is triggered, a model predictive control (MPC) is carried out in the second step for scheduling using the expected value of system information. By exercising this policy with different thresholds, one can explore the trade-offs between short-term benefits and battery lifetime, and identify an optimal threshold that maximizes the total economic benefits within the battery lifetime. An evaluation study in a real-world HES project is presented to illustrate the proposed framework. Compared with traditional optimal dispatch algorithms, the proposed method can significantly improve the economic benefits of an HES scheduled under forecast uncertainties.
Ma, Xu↗
Combined optimal control and estimation.
Combined optimization problem, equivalent to dual control problem, considering determination of optimal control policies for plant under random disturbances, using iterative equations
CONTROL SYSTEM↗
Sensitivity of joint contagiousness and susceptibility-based dynamic optimal control strategies for HIV prevention
Predicting the population-level effects of an infectious disease intervention that incorporate multiple modes of intervention is complicated by the joint non-linear dynamics of both infection transmission and the intervention itself. In this paper, we consider the sensitivity of Dynamic Optimal Control Profiles (DOCPs) for the optimal joint investment in both a contagiousness and susceptibility-based control of HIV to bio-behavioral, economic, and programmatic assumptions. The DOCP is calculated using recently developed numerical algorithms that allow controls to be represented by a set of piecewise constant functions that maintain a constant yearly budget. Our transmission model assumes multiple stages of HIV infection corresponding to acute and chronic infection and both within- and between-individual behavioral heterogeneity. We parameterize a baseline scenario from a longitudinal study of sexual behavior in MSM and consider sensitivity of the DOCPs to deviations from that baseline scenario. In the baseline scenario, the primary determinant of the dominant control were programmatic factors, regardless of budget. In sensitivity analyses, the qualitative aspects of the optimal control policy were often robust to significant deviation in assumptions regarding transmission dynamics. In addition, we found several conditions in which long-term joint investment in both interventions was optimal. Our results suggest that modeling in the service of decision support for intervention design can improve population-level effects of a limited set of economic resources. We found that economic and programmatic factors were as important as the inherent transmission dynamics in determining population-level intervention effects. Given our finding that the DOCPs were robust to alternative biological and behavioral assumptions it may be possible to identify DOCPs even when the data are not sufficient to identify a transmission model.
59 BASIC BIOLOGICAL SCIENCES↗