SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
New algorithm for quadratic programming problems with application to control, determining input requiring minimum time from initial state to target state
Incorporating physics information into the deep reinforcement learning (DRL) process is a promising approach for addressing the challenges faced in learning-based control design problems for physical systems. Power grid dynamics, being a physical system, adheres to specific physical laws, constraints, as well as operational and control rules. Therefore, consideration of such physics-based law improves the learning process drastically. In general, traditional grid control schemes rely on rule-based mechanisms that cannot adapt to changing operating conditions. To improve the adaptability and computation time, recent research has seen a surge of DRL-based applications in power grid control. A generic DRL-based control design imposes the system performance requirements through the design of reward functions. In some cases, some of the important physics information is injected through this reward function. However, due to the complex dynamics and large state-action space, learning an optimal DRL policy often becomes challenging. Inspired by the latest developments in general machine learning (ML) research, power system researchers have been investigating more direct ways of incorporating physics knowledge into DRL training. This chapter specifically focuses on these aspects of physics-informed DRL designs in grid control. It discusses the significance, applications, research gaps, and open problems that need to be addressed in future research.
With large-scale distributed photovoltaics (PVs) being integrated into distribution networks (DNs), coordinated optimal power flow (OPF) of high voltage (HV) and medium voltage (MV) DNs should be investigated to optimally dispatch the distributed PVs and other network devices. Here, this paper presents a hierarchical distributed OPF method for HV and MV DNs with on-load tap changers, reactive power compensators, feeder switches and distributed PVs. A hierarchical master-slave control architecture is applied to implement coordinated OPF of two-layer DNs. The HV master problem and MV subproblems are transformed into mixed-integer convex problems respectively with second order cone programming and LinDistFlow approximation. Since there is no efficient distributed algorithm to solve such OPF models with integer subproblems, a novel distributed algorithm is proposed in this paper to efficiently solve the hierarchical coordinated OPF model with integer subproblems in a distributed manner. In the proposed algorithm, the coordinated OPF model is solved in a branch-and-bound framework, where in each branch node generalized Benders decomposition (GBD) algorithm is applied to decompose the coordinated OPF model into a master problem and relaxed subproblems and solves them iteratively to get optimal solution. The GBD optimal and feasible cutting planes generated in a branch node are proved to be valid for its descendants. Moreover, three acceleration techniques are introduced into the proposed algorithm to improve computational efficiency. Finally, the effectiveness and accuracy of the proposed method are verified via simulation tests in Jinzhai DNs of China.
Multi Model Monte Carlo with Python (\mxmc {}) is a software package developed as a general capability for computing the statistics of outputs from an expensive, high-fidelity model by leveraging faster, low-fidelity models for speedup. Motivated by uncertainty propagation problems where classical Monte Carlo (MC) simulation is computationally intractable, various multi-model MC approaches have recently emerged that yield unbiased estimators with significantly reduced variance relative to MC for the same cost. These existing methods include multi-level Monte Carlo (MLMC), multi-fidelity Monte Carlo (MFMC), and approximate control variates (ACV). Given a fixed computational budget and a collection of models with varying cost/accuracy, each method seeks a sample allocation strategy across the models that results in an estimator with optimal variance reduction. \mxmc {} is a versatile tool that enables convenient access to many existing multi-model MC approaches within one modular and extensible package. With \mxmc {}, users can easily compare existing methods to determine the best choice for their particular problem, while developers have a basis for implementing and sharing new variance reduction approaches. This report introduces the \mxmc {} software, providing a summary of the problem-solving workflow for users as well as a brief overview of the code layout for developers.
Explicit energy minimization objectives are often discouraged in signal optimization algorithms due to its negative impact on mobility performance. One potential direction to solve this problem is to provide a balanced objective function to achieve desired mobility with minimized energy consumption. This research developed a reinforcement learning (RL) based control with reward functions considering energy and mobility in a joint manner-a penalty function is introduced for number of stops. Further, we proposed a clustering-based technique to make the state-space finite which is critical for a tractable implementation of the RL algorithm. We implemented the algorithm in a calibrated NG-SIM network within a traffic micro-simulator-PTV VISSIM. With sole focus on energy, we report 47% reduction in energy consumption when compared with existing signal control schemes, however causing a 65.6% increase in system travel time. In contrast, the control strategy focusing on energy minimization with penalty for stops yields 6.7% reduction in energy consumption with 27% increase in system travel time. The developed RL algorithm with a flexible penalty function in the reward will achieve desired energy goals for a network of signalized intersections without compromising on the mobility performance.
Rapid penetration of renewable energy based Distributed Energy Resources (DER) has the potential to exacerbate the challenges inherent in grid frequency and voltage regulation. However, their real time controllability can be leveraged to not only mitigate such challenges, but improve the economics and quality of grid operations. A plethora of DER scheduling algorithms have been developed for fast and efficient scheduling of these resources. These algorithms schedule the complex - real and reactive power injected into (or drawn from) the grid by the DERs under various cost objective functions and grid constraints. However, a vast majority of such algorithms assume the feasible range of power injections to be continuous. In reality, the control options available for several DERs form a discrete space. This implies that the feasible range of power injections also forms a discrete space. This makes DER scheduling an NP-hard problem. Integer/Mixed Integer program based algorithms, which guarantee optimality in the objective value, have been developed. But these are prohibitively expensive in terms of computational time requirements. On the other hand, fast heuristic based solutions have also been developed. But as they do not provide any guarantee on the optimality of the objective, or on satisfying the constraints, they are unreliable for grid operations. Recent works have developed approximation algorithms which offer the best of both the approaches above: (near) optimality and low computational complexity, both of which are governed by a user defined accuracy parameter ϵ. In this work, we significantly improve upon such existing works by developing an approximation algorithm which schedules DERs with complex injections (as opposed to only real injection) and whose runtime is polynomial (as opposed to exponential) in 1/ϵ. We provide theoretical analysis and support them with experimental results to validate the guarantees provided by our approximation algorithm.
In today's competitive environment, both industry and government agencies are under enormous pressure to reduce the time and cost of multidisciplinary design projects. A number of frameworks have been introduced to assist in this process by facilitating the integration of and communication among diverse disciplinary codes. An examination of current frameworks reveals weaknesses in various areas such as sequencing, displaying, monitoring, and controlling the design process. The objective of this research is to explore how Web technology, in conjunction with an existing framework, can improve these areas of weakness. This paper describes a system that executes a sequence of programs, monitors and controls the design process through a Web-based interface, and visualizes intermediate and final results through the use of Java(Tm) applets. A small sample problem, which includes nine processes with two analysis programs that are coupled to an optimizer, is used to demonstrate the feasibility of this approach.
A straightforward extension of a solution to the decentralized linear-Quadratic-Gaussian problem is proposed that allows its use for commonly encountered classes of problems that are currently solved with the extended Kalman filter. This extension allows the system to be partitioned in such a way as to exclude the nonlinearities from the essential algebraic relationships that allow the estimation and control to be optimally decentralized.
This research continued work that began under the support of NASA Grant NAG1-2119. The focus of this effort was to continue investigations of Genetic Algorithm (GA) approaches that could be used to solve an actuator placement problem by treating this as a discrete optimization problem. In these efforts, the actuators are assumed to be "smart" devices that change the aerodynamic shape of an aircraft wing to alter the flow past the wing, and, as a result, provide aerodynamic moments that could provide flight control. The earlier work investigated issued for the problem statement, developed the appropriate actuator modeling, recognized the importance of symmetry for this problem, modified the aerodynamic analysis routine for more efficient use with the genetic algorithm, and began a problem size study to measure the impact of increasing problem complexity. The research discussed in this final summary further investigated the problem statement to provide a "combined moment" problem statement to simultaneously address roll, pitch and yaw. Investigations of problem size using this new problem statement provided insight into performance of the GA as the number of possible actuator locations increased. Where previous investigations utilized a simple wing model to develop the GA approach for actuator placement, this research culminated with application of the GA approach to a high-altitude unmanned aerial vehicle concept to demonstrate that the approach is valid for an aircraft configuration.
This paper presents a novel approach for automated generation of fully connected finite-element meshes for all internal structural components and skins of a given wing-body geometry model, controlled by a few conceptual-level structural layout parameters. Internal structural components include spars, ribs, frames, and bulkheads. Structural layout parameters include spar/rib locations in wing chordwise/spanwise direction and frame/bulkhead locations in longitudinal direction. A simple shell thickness optimization problem with two load conditions is used to verify versatility and robustness of the automated meshing process. The automation process is implemented in ModelCenter starting from an OpenVSP geometry and ending with a NASTRAN 200 solution. One subsonic configuration and one supersonic configuration are used for numerical verification. Two different structural layouts are constructed for each configuration and five finite-element meshes of different sizes are generated for each layout. The paper includes various comparisons of solutions of 20 thickness optimization problems, as well as discussions on how the optimal solutions are affected by the stress constraint bound and the initial guess of design variables.
A preprocessing method to correct for image degradation is proposed which can be thought of as a generalization and extension of previous work by Smith (1966) and Stuller (1972). This method accomodates the problem of noncircularly symmetric imaging system point-spread functions, provides for controlled extent of the preprocessing filter to minimize distortion due to transients resulting from truncation errors and edge effects, can be used with various kinds of system noise, and can be readily extended to provide constraint of other system parameters. The analysis relates to a line-scanner system, although it is applicable in principle to many other system configurations.
The ability of nuclear reactors to operate their power conversion cycles more flexibly will enhance their value to energy grids with variable pricing. Current nuclear control systems are typically classical controllers that are often based on proportional-integral-derivative (PID) control. This paper presents a method of augmenting the existing PID control for difficult transient operations in nuclear power plants using a reinforcement learning–derived feedforward signal applied in real time. The agents, which are trained on a test thermal load-following problem, are designed to improve steam generator outlet temperature control for a range of fast load-following scenarios covering ramp rates from 9%/min to 15%/min. Several reinforcement learning algorithms were initially investigated for the training of the feedforward agents with deep Q-learning (DQN) and proximal policy optimization (PPO) networks, which were found to be the most promising. The DQN controllers utilize discrete actions, giving them a better disturbance rejection at steady state but inconsistent response to initial temperature deviations. In contrast, PPO-trained agents, which take continuous actions except for a dead zone around zero, were shown to have the best combination of high disturbance rejection at steady state and good tracking of the desired temperature value. The ability of the PPO agent was also examined, with the average time of decision making found to be on the order of 1 ms. The fault properties of the controller under the loss of the reinforcement learning agent feedforward signal were also examined. The controller showed strong performance in situations of “no-signal” faults. but was less good at handling “stuck-at” faults, where the feedforward signal remains at a set value. In both cases, however, the PID was able to successfully maintain stability, eventually returning the system to a steady state. It is hoped that this work will allow for the proposed control architecture to be examined for more difficult control problems such that it may eventually be used to adapt existing nuclear plants for more aggressive load-following on grids of the future.
Information asymmetry between the Distribution System Operator (DSO) and Distributed Energy Resource Aggregators (DERAs) obstructs designing effective incentives for voltage regulation. To capture this effect, we employ a Stackelberg game-theoretic framework, where the DSO seeks to overcome the information asymmetry and refine its incentive strategies by learning from DERA behavior over multiple iterations. We introduce a model-based online learning algorithm for the DSO, aimed at inferring the relationship between incentives and DERA responses. Given the uncertain nature of these responses, we also propose a distributionally robust incentive design model to control the probability of voltage regulation failure and then reformulate it into a convex problem. This model allows the DSO to periodically revise distribution assumptions on uncertain parameters in the decision model of the DERA. Finally, we present a gradient-based method that permits the DSO to adaptively modify its conservativeness level, measured by the size of a Wasserstein metric-based ambiguity set, according to historical voltage regulation performance. The effectiveness of our proposed method is demonstrated through numerical experiments.
The primary goal of this paper is to provide an efficient solution algorithm based on the augmented Lagrangian framework for optimization problems with a stochastic objective function and deterministic constraints. Our main contribution is combining the augmented Lagrangian framework with adaptive sampling, resulting in an efficient optimization methodology validated with practical examples. To achieve the presented efficiency, here we consider inexact solutions for the augmented Lagrangian subproblems, and through an adaptive sampling mechanism, we control the variance in the gradient estimates. Furthermore, we analyze the theoretical performance of the proposed scheme by showing equivalence to a gradient descent algorithm on a Moreau envelope function, and we prove sublinear convergence for convex objectives and linear convergence for strongly convex objectives with affine equality constraints. The worst-case sample complexity of the resulting algorithm, for an arbitrary choice of penalty parameter in the augmented Lagrangian function, is $\mathscr{O}$(ϵ -3-δ ) , where ϵ > 0 is the expected error of the solution and δ > 0 is a user-defined parameter. If the penalty parameter is chosen to be $\mathscr{O}$(ϵ -1 ), we demonstrate that the result can be improved to $\mathscr{O}$(ϵ -2 ) , which is competitive with the other methods employed in the literature. Moreover, if the objective function is strongly convex with affine equality constraints, we obtain $\mathscr{O}$(ϵ -1 log(1/ϵ)) complexity. Finally, we empirically verify the performance of our adaptive sampling augmented Lagrangian framework in machine learning optimization and engineering design problems, including topology optimization of a heat sink with environmental uncertainty.
One of the most effective first-order algorithms for solving trajectory optimization problems is the sequential gradient-restoration algorithm (SGRA). Originally developed in the primal formulation, this algorithm is extended to incorporate a dual formulation. Both the primal formulation and the dual formulation involve a sequence of two-phase cycles, each cycle including a gradient phase and a restoration phase. In turn, each iteration of the gradient phase and the restoration phase requires the solution of an auxiliary minimization problem (AMP). In the primal formulation, the AMP is solved with respect to the variations of the state, the control, and the parameter. In the dual formulation, the AMP is solved with respect to the Lagrange multipliers. A characteristic of the dual formulation is that the AMPs associated with the gradient phase and the restoration phase of SGRA can be reduced to mathematical programming problems involving a finite number of parameters as unknowns. A comparison of the primal formulation and the dual formulation is presented. The comparison is done in terms of several trajectory optimization problems having current aerospace interest.
The ever-growing higher penetration of distributed energy resources (DERs) in low-voltage (LV) distribution systems brings both opportunities and challenges to voltage support and regulation. This paper proposes a deep reinforcement learning (DRL)-based scheduling scheme of energy storage systems (ESSs) to mitigate system voltage deviations in unbalanced LV distribution networks. The ESS-based voltage regulation problem is formulated as a multi-stage quadratic stochastic program, with the objective of minimizing the expected total daily voltage regulation cost while satisfying operational constraints. While existing voltage regulation methods are mostly focused on onetime- step control, this paper explores a day-horizon systemwide voltage regulation problem. In other words, the size of action and state spaces are extremely high-dimensional and need to be delicately handled. Furthermore, in order to overcome the difficulty of modeling uncertainties and develop a realtime solution, a learn-to-schedule feedback control framework is proposed by adapting the problem to a model-free DRL setting. The proposed algorithm is tested on a customized 6-bus system and a modified IEEE 34-bus system. Simulation results validate the effectiveness and near-optimality of voltage regulation by ESS in comparison with a deterministic quadratic program solution.
The accelerating deployment of solar photovoltaics into low-voltage distribution networks can cause reverse power flow and overvoltage problems. However, if coordinated properly, the real and reactive power flexibility of these resources enables distribution operators to manage their networks more efficiently. Existing literature is rich in droop-based control (Volt-Watt and Volt-VAr) and optimization-based distributed energy coordination for four-quadrant control of photovoltaics to prevent overvoltage issues. While optimal coordination can effectively mitigate overvoltage, it tends to treat resources at sensitive parts of the grid unfairly. Here, to address this concern, we propose a distributed optimal power flow formulation that incorporates fairness in curtailing photovoltaic generation and utilizes the reactive power capability of smart inverters. The proposed distributed formulation allows for scalable resource aggregation that can be leveraged to achieve fairness within a certain segment of the grid and/or fairness across the entire network. Fair curtailment of photovoltaic systems is demonstrated with aggregation at each of two layers in a distribution network: 1) area-level fairness and 2) feeder-level fairness. To explore the trade-off between fairness and optimal utilization, the fairness-aware control actions are compared against the performance of a centralized controller that aims to maximize the aggregate PV generation without incorporating fairness. Simulation results show that introducing area-level fairness increased curtailment by 0.0101 percentage points and feeder-level fairness increased curtailment by 0.0458 percentage points compared to a fairness-agnostic control.
The relatively low values of lift-curve slope produced by highly swept arrow wings, coupled with the low scrape angle of the fuselage, resulted in relatively low values of take-off and approach lift coefficients. Through the use of more efficient high-lift systems and the application of propulsive-lift concepts, it is possible to optimize the engine-airframe design for maximum range potential and also to provide good low-speed performance. Nose strakes provide significant improvements in directional stability characteristics and the use of a propulsive lateral control system provides a solution to problems associated with inherently low levels of lateral control.