Search NASA⌕ Search

SEARCH · Search NASA

Results for “linear quadratic regulator (LQR) control”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

13 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Stability-Constrained Learning for Frequency Regulation in Power Grids With Variable Inertia

The increasing penetration of converter-based renewable generation has resulted in faster frequency dynamics, and low and variable inertia. As a result, there is a need for frequency control methods that are able to stabilize a disturbance in the power system at timescales comparable to the fast converter dynamics. This paper proposes a combined linear and neural network controller for inverter-based primary frequency control that is stable at time-varying levels of inertia. We model the time-variance in inertia via a switched affine hybrid system model. We derive stability certificates for the proposed controller via a quadratic candidate Lyapunov function. We test the proposed control on a 12-bus 3-area test network, and compare its performance with a base case linear controller, optimized linear controller, and finite-horizon Linear Quadratic Regulator (LQR). Our proposed controller achieves faster mean settling time and over 50% reduction in average control cost across 100 inertia scenarios compared to the optimized linear controller. Unlike LQR which requires complete knowledge of the inertia trajectories and system dynamics over the entire control time horizon, our proposed controller is real-time tractable, and achieves comparable performance to LQR.

data-driven control↗
DOE OSTI2021

Robust Output Feedback Control Design for Inertia Emulation by Wind Turbine Generators

Wind generation has gained widespread use as a renewable energy source. Most wind turbines and other renewables connected to the grid through converters result in a reduction in the natural inertial response to grid frequency changes. The doubly-fed induction generator (DFIG) can be controlled to compensate for this reduction and, in fact, provide faster response than traditional synchronous machines. This paper proposes to design observer based output feedback linear quadratic regulator (LQR) and H control laws to realize the inertia emulation function and deliver fast frequency support. Furthermore, the aim is to track the reference speed by a diesel synchronous generator (DSG) in order to reach the desired inertia. The control signal is computed based on a reduced order model using the balanced truncation technique. A comparison with selective modal analysis (SMA) and balanced truncation model reduction techniques is presented. Comprehensive results show the effective emulation of synthetic inertia by implementing the control laws on a nonlinear three- phase diesel-wind system. The proposed technique is analyzed for different short circuit ratio (SCR) scenarios.

17 WIND ENERGY↗
DOE OSTI2020

Optimizing Signal Timing Control for Large Urban Traffic Networks Using an Adaptive Linear Quadratic Regulator Control Strategy

We discuss how traffic signal control is important for intersection safety and efficiency. However, most traffic signal control methods are designed for individual intersections or corridors. Although some adaptive control systems have been developed, the methods used are often proprietary and not published, making it difficult to evaluate their effectiveness. This study proposes an adaptive multi-input and multi-output traffic signal control method that not only can improve network-wide traffic operations in terms of reduced traffic delay and energy consumption, but also is more computationally feasible than existing centralized signal control methods. Considering intersection interactions, a linear dynamic traffic system model was built and adaptively updated to reflect how the signal control input of each intersection affects network-wide vehicle travel delay. Based on the system model, an adaptive linear-quadratic regulator (LQR) was designed to minimize both traffic delay and incremental changes in the control input. The proposed control method was evaluated in a microscopic traffic simulation environment with a 35-intersection network of Bellevue City, Washington. Simulation results show that the proposed method had shorter average traffic delays in the network when compared with the traffic delays controlled by the state-of-the-art max-pressure, self-organizing traffic lights, and independent deep Q network methods.

42 ENGINEERING↗
DOE OSTI2022

Adaptive Urban Traffic Signal Control for Multiple Intersections: An LQR Approach

Traffic congestion leads to severe problems especially in urban traffic networks. It increases the chance of accidents, energy waste, and social costs. In order to address these problems, an adaptive linear quadratic regulator (LQR) approach is developed for traffic signal control at multiple intersections in an urban area. The proposed method controls the green time of the traffic signals to reduce traffic congestion and smooth traffic flow. Real-world data from vision-based traffic sensors are used to build the traffic network model, which mimics the real-world traffic behavior. In addition, the proposed control utilizes recursive least square parameter estimation, which is capable of tracking dynamic changes in traffic conditions. Simulation of Urban MObility (SUMO) is used to analyze the efficacy of the proposed method. Results of the simulation show that the proposed method outperforms pretimed control in various aspects.

adaptive LQR control↗
DOE OSTI2023

Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

This paper delves into designing feedback control gains for a continuous-time linear quadratic regulator (LQR) problem that is constrained to certain predefined structure with unknown state matrix. We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees in order to design these structured gains using the trajectory measurements of states and controls. Here we first formulate a model-based framework using dynamic programming (DP) to embed the structural constraint to the LQR gain computation in the continuous-time setting, and then subsequently, formulate a policy iteration RL algorithm that can alleviate the requirement of known state matrix in conjunction with maintaining the feedback gain structure. The design enables a distributed learning control design which is necessary for many large-scale cyber-physical systems. Theoretical guarantees are provided for stability and convergence of the structured reinforcement learning (SRL) algorithm. We validate our theoretical results with numerical simulations on a multi-agent networked linear time-invariant (LTI) dynamic system.

42 ENGINEERING↗
DOE OSTI2020

An Optimal Power Control Strategy for Grid-Following Inverters in a Synchronous Frame

This work proposes a power control strategy based on the linear quadratic regulator with optimal reference tracking (LQR-ORT) for a three-phase inverter-based generator (IBG) using an LCL filter. The use of an LQR-ORT controller increases robustness margins and reduces the quadratic value of the power error and control inputs during transient response. A model in a synchronous reference frame that integrates power sharing and voltage–current (V–I) dynamics is also proposed. This model allows for analyzing closed-loop eigenvalue location and robustness margins. The proposed controller was compared against a classical droop approach using proportional-resonant controllers for the inner loops. Mathematical analysis and hardware-in-the-loop (HIL) experiments under variations in the LCL filter components demonstrate fulfillment of robustness and performance bounds of the LQR-ORT controller. Experimental results demonstrate accuracy of the proposed model and the effectiveness of the LQR-ORT controller in improving transient response, robustness, and power decoupling.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Data-Driven Optimal Control of Traffic Signals for Urban Road Networks

This paper studies the issue of data-driven optimal control design for traffic signals of oversaturated urban road networks. The signal control system based on the store and forward model is generally uncontrollable for which the controllable decomposition is needed. Instead of identifying the unknown parameters like saturation rates and turning ratios, a finite number of measured trajectories can be used to parametrize the system and help directly construct a transformation matrix for Kalman controllable decomposition through the fundamental lemma of J. C. Willems. On top of that, an infinite-horizon linear quadratic regulator (LQR) problem is formulated considering the constraints of green times for traffic signals. The problem can be solved through a two-phase data-driven learning process, where one solves an infinite-horizon unconstrained LQR problem and the other solves a finite-horizon constrained LQR problem. The simulation result shows the theoretical analysis is effective and the proposed data-driven controller can yield desired performance for reducing traffic congestion.

Liu, Tong↗
DOE OSTI2021

A Secure Learning Control Strategy via Dynamic Camouflaging for Unknown Dynamical Systems under Attacks

This paper presents a secure reinforcement learning (RL) based control method for unknown linear time-invariant cyber-physical systems (CPSs) that are subjected to compositional attacks such as eavesdropping and covert attack. We consider the attack scenario where the attacker learns about the dynamic model during the exploration phase of the learning conducted by the designer to learn a linear quadratic regulator (LQR), and thereafter, use such information to conduct a covert attack on the dynamic system, which we refer to as doubly learning-based control and attack (DLCA) framework. We propose a dynamic camouflaging based attack-resilient reinforcement learning (ARRL) algorithm which can learn the desired optimal controller for the dynamic system, and at the same time, can inject sufficient misinformation in the estimation of system dynamics by the attacker. The algorithm is accompanied by theoretical guarantees and extensive numerical experiments on a consensus multi-agent system and on a benchmark power grid model.

Mukherjee, Sayak↗
DOE OSTI2020

Data-driven modeling and control of dynamical systems using Koopman and Perron-Frobenius operators

This dissertation studies the data-driven modeling and control problem of nonlinear systems by exploiting the linear operator theoretic framework involving Koopman and Perro-Frobenius operator. A systematic linear-operator based controller design procedure has been established, which can be used to solve a variety of nonlinear control problems, including feedback stabilization using control Lyapunov functions, optimal quadratic regulation using Koopman eigenfunctions and convex optimization formulation of optimal control problem using P-F and Koopman operator approximation. As the core of data-driven modeling, we first propose a new algorithm for the finite-dimensional approximation of the linear transfer Koopman and Perron-Frobenius operator from time-series data. We argue that the existing approach for the finite-dimensional approximation of these transfer operators such as Dynamic Mode Decomposition (DMD) and Extended Dynamic Mode Decomposition (EDMD) do not capture two important properties of these operators, namely positivity and Markov property. The algorithm we propose preserves these two properties. We call the proposed algorithm as naturally structured DMD (NSDMD) since it retains the inherent properties of these operators. Naturally structured DMD algorithm leads to a better approximation of the steady-state dynamics of the system regarding computing Koopman and Perron- Frobenius operator eigenfunctions and eigenvalues. However, preserving positivity property is critical for capturing the real transient dynamics of the system. This positivity property of the transfer operators and it's finite-dimensional approximation play an important role for controller and estimator design of nonlinear systems. To solve the feedback stabilization problem for nonlinear control systems, we tried to take advantage of the Koopman operator framework. The Koopman operator approach provides a linear representation for a nonlinear dynamical system and a bilinear representation for a nonlinear control system. The problem of feedback stabilization of a nonlinear control system is then transformed to the stabilization of a bilinear control system. We propose a control Lyapunov function (CLF)-based approach for the design of stabilizing feedback controllers for the bilinear system. The search for finding a CLF for the bilinear control system is formulated as a convex optimization problem. This leads to a schematic procedure for designing CLF-based stabilizing feedback controllers for the bilinear system and hence the original nonlinear system. Another advantage of the proposed controller design approach outlined in this dissertation is that it does not require explicit knowledge of system dynamics. In particular, the bilinear representation of a nonlinear control system in the Koopman eigenfunction space can be obtained from time-series data. Next, we study the optimal quadratic regulation problem for nonlinear systems. The linear operator theoretic framework involving the Koopman operator is used to lift the dynamics of nonlinear control system to an infinite-dimensional bilinear system. The optimal quadratic regulation problem for nonlinear system is formulated in terms of the finite-dimensional approximation of the bilinear system. A convex optimization-based approach is proposed for solving the quadratic regulator problem for bilinear system. We applied a variety of examples and compared the simulation results between our framework and conventional LQR control using linearized model. For more general optimal control problems, we provide a density-function based convex formulation for the optimal control problem of the nonlinear system. The convex formulation relies on the duality result in the stability theory of a dynamical system involving density function and Perron-Frobenius operator. The optimal control problem is formulated as an infinite-dimensional convex optimization program. The finite-dimensional approximation of the optimization problem relies on the recent advances made in the data-driven computation of the Koopman operator, which is dual to the Perron-Frobenius operator. Simulation results are presented to demonstrate the application of the developed framework.

Huang, Bowen↗
DOE OSTI2024

Leveraging Multitime Hamilton–Jacobi PDEs for Certain Scientific Machine Learning Problems

Hamilton-Jacobi partial differential equations (HJ PDEs) have deep connections with a wide range of fields, including optimal control, differential games, and imaging sciences. By considering the time variable to be a higher dimensional quantity, HJ PDEs can be extended to the multi-time case. In this paper, we establish a novel theoretical connection between specific optimization problems arising in machine learning and the multi-time Hopf formula, which corresponds to a representation of the solution to certain multi-time HJ PDEs. Through this connection, we increase the interpretability of the training process of certain machine learning applications by showing that when we solve these learning problems, we also solve a multi-time HJ PDE and, by extension, its corresponding optimal control problem. As a first exploration of this connection, we develop the relation between the regularized linear regression problem and the Linear Quadratic Regulator (LQR). We then leverage our theoretical connection to adapt standard LQR solvers (namely, those based on the Riccati ordinary differential equations) to design new training approaches for machine learning. Lastly, we provide some numerical examples that demonstrate the versatility and possible computational advantages of our Riccati-based approach in the context of continual learning, post-training calibration, transfer learning, and sparse dynamics identification.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Risk-Constrained Reinforcement Learning for Inverter-Dominated Power System Controls

Here, this paper develops a risk-aware controller for grid-forming inverters (GFMs) to minimize large frequency oscillations in GFM inverter-dominated power systems. To tackle the high variability from loads/renewables, we incorporate a mean-variance risk constraint into the classical linear quadratic regulator (LQR) formulation for this problem. The risk constraint aims to bound the time-averaged cost of state variability and thus can improve the worst-case performance for large disturbances. The resulting risk-constrained LQR problem is solved through the dual reformulation to a minimax problem, by using a reinforcement learning (RL) method termed as stochastic gradient-descent with max-oracle (SGDmax). In particular, the zero-order policy gradient (ZOPG) approach is used to simplify the gradient estimation using simulated system trajectories. Numerical tests conducted on the IEEE 68-bus system have validated the convergence of our proposed SGDmax for GFM model and corroborate the effectiveness of the risk constraint in improving the worst-case performance while reducing the variability of the overall control cost.

Frequency control↗
DOE OSTI2021

Model-based and Model-free Designs for an Extended Continuous-time LQR with Exogenous Inputs

We present an extended linear quadratic regulator (LQR) design for continuous-time linear time-invariant (LTI) systems in the presence of exogenous inputs. We first propose a model-based solution with cost minimization guarantees for states and inputs using dynamic programming (DP). The control law consists of a combination of the optimal state feedback and an additional optimal term dependent on the exogenous inputs. The control gains for the two components are obtained by solving a set of matrix differential equations. We provide these solutions for both finite horizons and steady-state cases. In the second part of the paper, we formulate a reinforcement learning (RL) based algorithm which does not need any model information except the input matrix, and can compute an approximate steady-state LQR gain using measurements of the states, the control inputs, and the exogenous inputs. Both model-based and data-driven optimal control algorithms are tested with a numerical example under different exogenous inputs showcasing the effectiveness of the designs.

Mukherjee, Sayak↗