SEARCH · Search NASA

Results for “Learning control”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Efficient and assured reinforcement learning-based building HVAC control with heterogeneous expert-guided training

Abstract Building heating, ventilation, and air conditioning (HVAC) systems account for nearly half of building energy consumption and $$20\%$$ of total energy consumption in the US. Their operation is also crucial for ensuring the physical and mental health of building occupants. Compared with traditional model-based HVAC control methods, the recent model-free deep reinforcement learning (DRL) based methods have shown good performance while do not require the development of detailed and costly physical models. However, these model-free DRL approaches often suffer from long training time to reach a good performance, which is a major obstacle for their practical deployment. In this work, we present a systematic approach to accelerate online reinforcement learning for HVAC control by taking full advantage of the knowledge from domain experts in various forms . Specifically, the algorithm stages include learning expert functions from existing abstract physical models and from historical data via offline reinforcement learning, integrating the expert functions with rule-based guidelines, conducting training guided by the integrated expert function and performing policy initialization from distilled expert function. Moreover, to ensure that the learned DRL-based HVAC controller can effectively keep room temperature within the comfortable range for occupants, we design a runtime shielding framework to reduce the temperature violation rate and incorporate the learned controller into it. Experimental results demonstrate up to 8.8 X speedup in DRL training from our approach over previous methods, with low temperature violation rate.

Xu, Shichao↗

DOE OSTI2025

Safe Reinforcement Learning-Based Transient Stability Control for Islanded Microgrids With Topology Reconfiguration

This paper proposes a safe reinforcement learning (RL)-based transient stability emergency control (TSEC) method for islanded microgrids. RL requires extensive interaction with the environment to learn control strategies, hence, a data-driven approach is used as a substitute for time-consuming time-domain simulation calculations. Deep sigma point processes (DSPP), which is a Gaussian process model, is utilized to predict the normal distribution of transient stability of microgrids and to construct a transient stability chance constraint. Reward-constrained policy optimization (RCPO) can simultaneously achieve objective prediction, policy learning, and constraint cost coefficient update across multiple timescales. RCPO interacts with the DSPP-based microgrid environment through a multi-process parallel manner, greatly increasing the training speed. Case studies on a real islanded microgrid demonstrate that the proposed method can efficiently and quickly obtain the optimal emergency control strategy while adhering to all hard constraints.

14 SOLAR ENERGY↗

DOE OSTI2022

Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning

As power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits, there is increasing risk of large outages. Thus, there is an imperative need to enhance grid emergency control to maintain system reliability and security. Towards this end, great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years. However, existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications. Here, in this paper, we mitigate these limitations by developing a novel deep meta-reinforcement learning (DMRL) algorithm. The DMRL combines the meta strategy optimization together with DRL, and trains policies modulated by a latent space that can quickly adapt to new scenarios. We test the developed DMRL algorithm on the IEEE 300-bus system. We demonstrate fast adaptation of the meta-trained DRL polices with latent variables to new operating conditions and scenarios using the proposed method, which achieves superior performance compared to the state-of-the-art DRL and model predictive control (MPC) methods.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2021

Distributed deep reinforcement learning for simulation control

Abstract Several applications in the scientific simulation of physical systems can be formulated as control/optimization problems. The computational models for such systems generally contain hyperparameters, which control solution fidelity and computational expense. The tuning of these parameters is non-trivial and the general approach is to manually ‘spot-check’ for good combinations. This is because optimal hyperparameter configuration search becomes intractable when the parameter space is large and when they may vary dynamically. To address this issue, we present a framework based on deep reinforcement learning (RL) to train a deep neural network agent that controls a model solve by varying parameters dynamically. First, we validate our RL framework for the problem of controlling chaos in chaotic systems by dynamically changing the parameters of the system. Subsequently, we illustrate the capabilities of our framework for accelerating the convergence of a steady-state computational fluid dynamics solver by automatically adjusting the relaxation factors of the discretized Navier–Stokes equations during run-time. The results indicate that the run-time control of the relaxation factors by the learned policy leads to a significant reduction in the number of iterations for convergence compared to the random selection of the relaxation factors. Our results point to potential benefits for learning adaptive hyperparameter learning strategies across different geometries and boundary conditions with implications for reduced computational campaign expenses 4 4 Data and codes available at https://github.com/Romit-Maulik/PAR-RL . .

42 ENGINEERING↗

DOE OSTI2023

Cyber threat assessment of machine learning driven autonomous control systems of nuclear power plants

We report advanced cyber-attacks against critical infrastructure and the energy sector are becoming more common. With the invention of autonomous control systems (ACS) within advanced nuclear reactor designs, system designers, reactor operators, and regulators must consider cybersecurity during the design and operational phases. This article provides a cyber threat assessment of machine learning (ML)-based digital twinning (DT) technologies in the context of advanced reactor ACS. A cyber–physical testbed was created to emulate nuclear reactor digital instrumentation and controls (I&C) and act as a basis for the ACS. The ACS was designed as two plant-level DTs predicting reactor malfunctions and determining control actions and two component-level DTs responsible for classifying component states and forecasting component inputs and outputs (I/O). Two duplicate ACS designs– one using a traditional ML framework and one using an automated ML (AutoML) framework– were created and tested against cyber-attacks on training data, real-time process data, and ML model architectures to determine their respective qualitative cyber-risk in terms of likelihood and impact. Both frameworks showed similar cyber-resilience against training, real-time, and ML architecture attacks, proving that neither is inherently more secure. Recommended safeguard and security measures are posed to system designers, reactor operators, and regulators to maintain the cybersecurity of ML-based DT technologies such as ACS, prompting a holistic view of shared responsibility for maintaining cyber-secure ML-based systems.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2022

PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control

This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2019

Investigating the Impact of Connected Vehicle Market Share on the Performance of Reinforcement-Learning Based Traffic Signal Control

We aim to understand and explore the performance of reinforcement learning based signal control algorithms in a mixed environment with less than 100% market share of connected and automated vehicles (CAVs). Within a simulation environment, we have considered partial connectivity—less than 100% market share of CAVs—in the network and investigated the impact on the performance of the signal control algorithm. Two test networks including a four-intersection arterial in Lankershim Boulevard, CA, and a portion of downtown Springfield, IL, with 20 intersections. The first network is calibrated in the micro-simulator PTV Vissim with the US DOT provided NGSIM datasets. The results provide insights regarding the impact of the connectivity and sensing technologies on the practical implementation of traffic signal control algorithms that leverage the data sharing capability of a connected environment. For scenarios with 40% or more market share of CAVs, we observed improvement in the performance metrics—travel time, queue time, and energy consumption—with the increase in market share. Results from our experiments do not indicate any clear trend when the networks have low (less than 40%) market share of CAVs. The higher standard deviations as obtained from the statistical analyses of the performance metrics at low market share may indicate the instability of the RL controller arising from the partial (if not zero) observability of the traffic states. Further, we have conducted simplified scenario analyses to explore the impact of the market share of battery electric vehicles (BEVs) on energy consumption due to the regenerative braking feature. Our results and findings will be the foundation for the future reinforcement learning based control algorithm development that accounts for partial connectivity—less than 100% CAV market share, and the presence of BEVs in a network of connected and automated signalized intersections.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2022

Learning Local Volt/VAR Controllers Toward Efficient Network Operation with Stability Guarantees: Preprint

This paper considers the problem of voltage regulation in distribution network. The primary motivation is to keep voltages within pre-assigned operating limits by commanding the reactive power output of distributed energy resources (DERs) deployed in the grid. We develop a framework for developing local Volt/Var control that comprises of two main steps. In the first, exploiting historical data and for each DER, we learn a function representing desirable equilibrium points for the power network. These points approximate solutions of an Optimal Power Flow problem. In the second, we propose a control scheme for steering the network towards these favorable configurations. Theoretical conditions are derived to formally guarantee the stability of the developed control scheme and numerical simulations illustrate the effectiveness of the proposed approach.

data-driven control↗

DOE OSTI2025

Fast Machine Learning for Quantum Control of Microwave Qudits on Edge Hardware

Quantum optimal control is a promising approach to improve the accuracy of quantum gates, but it relies on complex algorithms to determine the best control settings. CPU or GPU-based approaches often have delays that are too long to be applied in practice. It is paramount to have systems with extremely low delays to quickly and with high fidelity adjust quantum hardware settings, where fidelity is defined as overlap with a target quantum state. Here, we utilize machine learning (ML) models to determine control-pulse parameters for preparing Selective Number-dependent Arbitrary Phase (SNAP) gates in microwave cavity qudits, which are multi-level quantum systems that serve as elementary computation units for quantum computing. The methodology involves data generation using classical optimization techniques, ML model development, design space exploration, and quantization for hardware implementation. Our results demonstrate the efficacy of the proposed approach, with optimized models achieving low gate trace infidelity near $10^{-3}$ and efficient utilization of programmable logic resources.

Sanders, Flor [Columbia U.]↗

DOE OSTI2025

Real-Time Sea State Estimation for Wave Energy Converter Control via Machine Learning

Wave energy converters (WECs) harness the untapped power of ocean waves to generate renewable energy, offering a promising solution to sustainable energy. An optimal WEC control strategy is essential to maximize power capture that dynamically adjusts system parameters in response to rapidly changing sea states. This study presents a novel control approach that leverages neural networks to estimate sea states from onboard WEC measurements such as position, velocity, and force. Using a point absorber WEC device as a test platform, our proposed approach estimates sea states in real-time and subsequently adjusts PID controller gains to maximize energy extraction. Simulation results across diverse sea conditions demonstrate that our strategy eliminates the need for external wave monitoring equipment while maintaining power capture efficiency. The results show that our neural network-based control technique can improve power capture by 25.6% while significantly reducing system complexity. This approach offers a practical alternative for WEC deployments where direct wave measurements are either infeasible or cost prohibitive.

PIDcontrol↗

DOE OSTI2023

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Here this article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2025

Multi-Agent Hierarchical Deep Reinforcement Learning for HVAC Control With Flexible DERs

As electricity consumption in commercial and residential buildings continues to rise, reducing energy costs presents an increasing challenge. Heating, ventilating, and air-conditioning (HVAC) systems, which typically account for 40%-50% of a building's energy use, are prime targets for energy savings. Intelligent control of HVAC temperature through the exploitation of HVAC load flexibility brings significant potential to reduce energy consumption and electricity expenses. The nonlinear models of HVAC systems challenge traditional control methods, while the uncertainty introduced by HVAC load flexibility complicates distributed energy resource (DER) management using conventional optimal dispatch techniques. In response to these challenges, we propose a hierarchical multi-agent deep reinforcement learning (DRL) approach. The lower-level agents focus on balancing comfort and energy conservation, while the upper-level DRL agents optimize the use of DERs to reduce peak demand based on the control outcomes of the HVAC by the lower-level agents. Here, in the upper-level agents, we incorporate a multi-agent structure based on ensemble learning, which acts based on historical and current data without relying on precise load forecasting to address the delayed rewarding issue in DRL. This allows for the effective reduction of energy costs. The proposed method is tested using a real-world microgrid comprising 413 buildings in Southern California, and the results demonstrate that our approach can significantly reduce overall electricity bills while ensuring the comfort of consumers and residents.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems

Physics-informed machine learning (PIML) is a set of methods and tools that systematically integrate machine learning (ML) algorithms with physical constraints and abstract mathematical models developed in scientific and engineering domains. As opposed to purely data-driven methods, {PIML} models can be trained from additional information obtained by enforcing physical laws such as energy and mass conservation. More broadly, {PIML} models can include abstract properties and conditions such as stability, convexity, or invariance. The basic premise of {PIML} is that the integration of ML and physics can yield more effective, physically consistent, and data-efficient models. This paper aims to provide a tutorial-like overview of the recent advances in {PIML} for dynamical system modeling and control. Specifically, the paper covers an overview of the theory, fundamental concepts and methods, tools, and applications on topics of: 1) physics-informed learning for system identification; 2) physics-informed learning for control; 3) analysis and verification of {PIML} models; and 4) physics-informed digital twins. The paper is concluded with a perspective on open challenges and future research opportunities.

Nghiem, Truong↗

DOE OSTI2020

High precision control and deep learning-based corn stand counting algorithms for agricultural robot

This paper presents high precision control and deep learning-based corn stand counting algorithms for a low-cost, ultra-compact 3D printed and autonomous field robot for agricultural operations. Currently, plant traits, such as emergence rate, biomass, vigor, and stand counting, are measured manually. This is highly labor-intensive and prone to errors. The robot, termed TerraSentia, is designed to automate the measurement of plant traits for efficient phenotyping as an alternative to manual measurements. In this paper, we formulate a Nonlinear Moving Horizon Estimator that identifies key terrain parameters using onboard robot sensors and a learning-based Nonlinear Model Predictive Control that ensures high precision path tracking in the presence of unknown wheel-terrain interaction. Moreover, we develop a machine vision algorithm designed to enable an ultra-compact ground robot to count corn stands by driving through the fields autonomously. The algorithm leverages a deep network to detect corn plants in images, and a visual tracking model to re-identify detected objects at different time steps. We collected data from 53 corn plots in various fields for corn plants around 14 days after emergence (stage V3 - V4). The robot predictions have agreed well with the ground truth with C robot =1.02×C human -0.86 and a correlation coefficient R=0.96. The mean relative error given by the algorithm is -3.78%, and the standard deviation is 6.76%. These results indicate a first and significant step towards autonomous robot-based real-time phenotyping using low-cost, ultra-compact ground robots for corn and potentially other crops.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Evaluation of automated stability testing in machining through closed-loop control and Bayesian machine learning

Here, this paper describes a system for automated identification of the optimal stable cutting parameters in milling through Bayesian machine learning and closed-loop control. The closed-loop control system consists of a process monitoring architecture, an analysis framework, and a feedback mechanism. The analysis framework consists of a Bayesian machine learning algorithm that learns a stability map given test results. The learned stability map is used to select parameters for stability testing using an expected improvement in the material removal rate criterion. The test parameters are communicated to the machine controller to complete the test cut through a feedback mechanism. The test cuts were monitored using an audio signal; the stability of the test cut was determined by analyzing the frequency content of the audio signal. The test result was fed back to the Bayesian learning algorithm to complete the loop. Experimental results demonstrate that the system can identify the optimal stable parameters without information about the cutting force model or the structural dynamics. The system provides a low-cost method for optimal stable parameter identification in an industrial environment.

Chatter↗

DOE OSTI2024

Unsupervised Learning for Equitable DER Control: Preprint

In the context of managing distributed energy resources (DERs) within distribution networks (DNs), this work focuses on the task of developing local controllers. We propose an unsupervised learning framework to train functions that can closely approximate optimal power flow (OPF) solutions. The primary aim is to establish specific conditions under which these learned functions can collectively guide the network towards desired configurations asymptotically, leveraging an incremental control approach. The flexibility of the proposed methodology allows to integrate fairness-driven components into the cost function associated with the OPF problem. This addition seeks to mitigate power curtailment disparities among DERs, thereby promoting equitable power injections across the network. To demonstrate the effectiveness of the proposed approach, power flow simulations are conducted using the IEEE 37-bus feeder. The findings not only showcase the guaranteed system stability but also underscore its improved overall performance.

asymptotic stability↗

DOE OSTI2024

A safe reinforcement learning algorithm for supervisory control of power plants

Traditional control theory-based methods require tailored engineering for each system and constant fine-tuning. In power plant control, one often needs to obtain a precise representation of the system dynamics and carefully design the control scheme accordingly. Model-free Reinforcement learning (RL) has emerged as a promising solution for control tasks due to its ability to learn from trial-and-error interactions with the environment. It eliminates the need for explicitly modeling the environment’s dynamics, which is potentially inaccurate. However, the direct imposition of state constraints in power plant control raises challenges for standard RL methods. To address this, we propose a chance-constrained RL algorithm based on Proximal Policy Optimization for supervisory control. Our method employs Lagrangian relaxation to convert the constrained optimization problem into an unconstrained objective, where trainable Lagrange multipliers enforce the state constraints. In conclusion, our approach achieves the smallest distance of violation and violation rate in a load-follow maneuver for an advanced Nuclear Power Plant design.

constrained optimization↗

DOE OSTI2022

Learning Stochastic Parametric Differentiable Predictive Control Policies

We present a scalable unsupervised learning-based method for obtaining explicit control policies for model predictive control problems for stochastic linear systems with additive uncertainties subject to nonlinear chance constraints. We call the proposed method stochastic parametric differentiable predictive control (SP-DPC), which extends the recently proposed deterministic DPC policy optimization algorithm. We formulate the SP-DPC as a deterministic approximation to the stochastic parametric constrained optimal control problem via independent sampling of the problem's parameters and uncertainties. This formulation allows us to directly compute the policy gradients via automatic differentiation of the problem's value function, evaluated over sampled parameters and uncertainties. In particular, the computed expectation of the problem's value function is backpropagated through the finite-time closed-loop system rollouts parametrized by a known nominal system dynamics model and neural control policy. We also provide theoretical probabilistic guarantees on closed-loop stability and chance constraints satisfaction for systems controlled by learned neural policies. We demonstrate the computational efficiency and scalability of the proposed policy optimization algorithm in three numerical examples, including systems with a large number of states or subject to nonlinear constraints.

Drgona, Jan↗