Search NASA⌕ Search

SEARCH · Search NASA

Results for “Learning control”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Deep reinforcement learning control for co-optimizing energy consumption, thermal comfort, and indoor air quality in an office building

With the recent demand for decarbonization and energy efficiency, advanced HVAC control using Deep Reinforcement Learning (DRL) becomes a promising solution. Due to its flexible structures, DRL has been successful in energy reduction for many HVAC systems. However, only a few researches applied DRL agents to manage the entire central HVAC system and control multiple components in both the water loop and the air loop, owing to its complex system structures. Moreover, those researches have not extended their applications by incorporating the indoor air quality, especially both CO2 and PM2.5concentrations, on top of energy saving and thermal comfort, as achieving those objectives simultaneously can cause multiple control conflicts. What's more, DRL agents are usually trained on the simulation environment before deployment, so another challenge is to develop an accurate but relatively simple simulator. Therefore, we propose a DRL algorithm for a central HVAC system to co-optimize energy consumption, thermal comfort, indoor CO2 level, and indoor PM2.5 level in an office building. To train the controller, we also developed a hybrid simulator that decoupled the complex system into multiple simulation models, which are calibrated separately using laboratory test data. The hybrid simulator combined the dynamics of the HVAC system, the building envelope, as well as moisture, CO2, and particulate matter transfer. Three control algorithms (rule-based, MPC, and DRL) are developed, and their performances are evaluated on the hybrid simulator environment with a realistic scenario (i.e., with stochastic noises). The test results showed that, the DRL controller can save 21.4 % of energy compared to a rule-based controller, and has improved thermal comfort, reduced indoor CO2 concentration. The MPC controller showed an 18.6 % energy saving compared to the DRL controller, mainly due to savings from comfort and indoor air quality boundary violations caused by unmeasured disturbances, and it also highlights computational challenges in real-time control due to non-linear optimization. Finally, we provide the practical considerations for designing and implementing the DRL and MPC controllers based on their respective pros and cons.

Guo, Fangzhou↗
DOE OSTI2021

A Secure Learning Control Strategy via Dynamic Camouflaging for Unknown Dynamical Systems under Attacks

This paper presents a secure reinforcement learning (RL) based control method for unknown linear time-invariant cyber-physical systems (CPSs) that are subjected to compositional attacks such as eavesdropping and covert attack. We consider the attack scenario where the attacker learns about the dynamic model during the exploration phase of the learning conducted by the designer to learn a linear quadratic regulator (LQR), and thereafter, use such information to conduct a covert attack on the dynamic system, which we refer to as doubly learning-based control and attack (DLCA) framework. We propose a dynamic camouflaging based attack-resilient reinforcement learning (ARRL) algorithm which can learn the desired optimal controller for the dynamic system, and at the same time, can inject sufficient misinformation in the estimation of system dynamics by the attacker. The algorithm is accompanied by theoretical guarantees and extensive numerical experiments on a consensus multi-agent system and on a benchmark power grid model.

Mukherjee, Sayak↗
DOE OSTI2026

Adaptive Reinforcement Learning Control for Power Distribution in Multi-Output Resonant Converters

This paper presents an adaptive reinforcement learning (ARL)-based control framework for efficient power distribution in a multi-output resonant converter for UAV applications. The proposed system is based on a high-frequency isolated resonant architecture, where a single energy source supplies multiple propulsion loads through independently controlled output rectifiers, addressing the need for coordinated multi-motor power management. The ARL framework dynamically allocates output power by learning optimal phase-shift control actions under varying load demands and operating conditions. The agent autonomously determines control parameters that maximize conversion efficiency while ensuring accurate power sharing among multiple outputs. In addition, the proposed approach enables adaptive operation without requiring detailed system modeling or manual tuning. Experimental results demonstrate stable and efficient performance over a wide range of operating conditions, confirming the effectiveness and robustness of the learning-based control strategy for multi-output resonant converter system.

Asa, Erdem [ORNL] (ORCID:0000000190884812)↗
DOE OSTI2023

Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

In recent years, advanced control strategies based on Deep Reinforcement Learning (DRL) proved to be effective in optimizing the management of integrated energy systems in buildings, reducing energy costs and improving indoor comfort conditions when compared to traditional reactive controllers. However, the scalability and implementation of DRL controllers are still limited since they require a considerable amount of time before converging to a near-optimal solution. This issue is currently addressed in literature through the offline pre-training of the DRL agent. However this solution results in two main critical issues: (1) the need to develop a building surrogate model to perform the training task, and (2) the need to perform a fine-tuning process over several training episodes to obtain a near-optimal control policy. In this context, this paper introduces an Online Transfer Learning (OTL) strategy that exploits two knowledge-sharing techniques, weight-initialization and imitation learning, to transfer a DRL control policy from a source office building to various target buildings in a simulation environment coupling EnergyPlus and Python. A DRL controller based on discrete Soft Actor–Critic (SAC) is trained on the source building to manage the operation of a cooling system consisting of a chiller and a thermal storage. Several target buildings are defined to benchmark the performance of the OTL strategy with that of a Rule-Based Controller (RBC) and two DRL-based control strategies, deployed in offline and online fashion. The strategy adopted for OTL emulates the real world implementation with a simulation process by implementing the transferred DRL agent for a single episode in the target buildings. Target buildings have the same geometrical features and are served by the same energy system as the source building, but differ in terms of weather conditions, electricity price schedules, occupancy patterns, and building envelope efficiency levels. The results show that the OTL strategy can reduce the cumulated sum of temperature violations on average by 50% and 80% respectively when compared to RBC and online DRL while enhancing the energy system operation with electricity cost savings ranging between 20% and 40%. Furthermore, the OTL agent performs slightly worse than the offline DRL controller but it does not require any modeling effort and can be implemented directly on target buildings emulating a real-world implementation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Reinforcement Learning Control for Enhancing Marine Hydrokinetic Turbine Energy Generation

This paper proposes a reinforcement learning-based method to maximize power generation for a direct-drive marine hydrokinetic turbine. A high levelized cost of energy (LCOE) is preventative in the widespread adoption of many marine energy conversion technologies. A straightforward way to reduce LCOE is to increase conversion efficiency and ensure maximum energy generation. The proposed method utilizes a damping control methodology, varying applied generator torque via a linear relationship between the applied damping coefficient and rotor speed. A state-action-reward-state-action (SARSA) algorithm has been used to learn the optimal control action for a given flow velocity. The proposed SARSA methodology uses Gaussian radial basis functions to create a three-dimensional surface to estimate the relationship between damping coefficient, incoming flow velocity, and coefficient of power (C p ). Here, the SARSA algorithm was compared against a baseline optimal tip speed ratio controller over a year-long flow velocity case profile while considering the effects of biofouling on the turbine system, where the proposed RL method generated 0.92% more energy than the baseline.

Damp↗
DOE OSTI2020

Machine learning control of an elliptically bent hard x-ray mirror

This article showcases the high-resolution control of an elliptically bent hard X-ray mirror optics at the Advanced Photon Source. The mirror uses a compact laminar flexure bending mechanism to achieve elliptical shapes covering a large range of focal distances. An array of capacitive sensors are used as a surface profiler for in-situ monitoring of the mirror shape. Machine learning and control techniques were used to change the mirror shape and focus the incident X-ray at predefined focal planes. The mirror surface shape error can be controlled to be within 40 nm rms with high repeatability. This technique gives the capability to focus incident X-ray beam within a range of focal distances corresponding to shape deformation range of a mirror optics. This work would be beneficial for controlling similar adaptive optics for multiple adaptive optics systems.

adaptive optics↗
DOE OSTI2024

Coherency-Aware Learning Control of Inverter-Dominated Grids: A Distributed Risk-Constrained Approach

Here, this letter investigates the importance of integrating the coherency knowledge for designing controllers to dampen sustained oscillations in wide-area power networks with significant penetration of inverter-interfaced resources. Coherency is a fundamental property of power systems, where time-scale separation in frequency dynamics leads to clustered behavior among generators of different groups. Large-scale penetration of inverter-driven low inertia resources replacing conventional synchronous generators (SGs) can lead to perturbation in the coherent partitioning; hence, integrating such information is of utmost importance for oscillation control designs. We present the coherency-aware design of a distributed output feedback-based reinforcement learning method that additionally incorporates risk constraints to capture the uncertainties related to net-load fluctuations. The use of domain-aware coherency information has produced improved training and oscillation performance than the coherency-agnostic control design, hence proving to be effective in controller design. Finally, we validated the proposed method with numerical experiments on the benchmark IEEE 68-bus test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Damping inter-area oscillation using reinforcement learning controlled TCSC

Appropriate damping control of the inter-area oscillation would ensure the grid stability and maintain the tie-line power transfer capability. In this paper, we proposed a novel reinforcement learning (RL) based power oscillation damping (POD) controller that uses Thyristor Controlled Series Compensators (TCSC) to damp inter-area oscillations. By leveraging the unbiased gradient direction estimation of the natural evolution strategy (NES), we successfully regulated the power flows on the tie-lines and damped inter-area oscillations through dynamically modulating the inserted reactance of the TCSC. The proposed RL-based POD controller has been tested on both two-area four-machine system and North American Minni-WECC system. Extensive studies have demonstrated the excellent performance of the proposed RL-based TCSC POD controller in damping inter-area oscillations.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Reinforcement Learning Control for Buildings Co-Optimizing Energy, Comfort, and Indoor Air Quality: An Annual Assessment

Efficient control of Heating, Ventilation, and Air Conditioning (HVAC) systems is crucial for optimizing energy use and maintaining indoor comfort in buildings. Traditional control methods, such as PID control, cannot handle energy use trade-offs among multiple components in the building energy system at a supervisory level. Reinforcement learning (RL) presents a promising solution, offering adaptive and data-driven control strategies that optimize performance over time. However, RL also faces several challenges, including the conflicts encountered in co-optimizing energy savings, occupant comfort, and indoor air quality, and the requirement for extensive interactions with the environment in training. We proposed a flexible simulation platform that integrates a hybrid model for RL training and designed an RL agent to control the entire central HVAC system, focusing on co-optimizing energy consumption, thermal comfort, and indoor air quality ($\text{CO}_{2}$ and PM2.5 concentrations). Finally, we evaluated the RL agent's performance over an annual cycle. Our findings indicate that the RL agent can effectively manage the HVAC system with 14.7 % energy savings annually and balance multiple objectives, which demonstrates significant potential for improving HVAC system control and sustainability in buildings.

Guo, Fangzhou↗
DOE OSTI2023

Fast Frequency Response using Reinforcement Learning-Controlled Wind Turbines

To fulfill the auxiliary grid services such as load regulation, spin and non-spin reserve, and frequency support during emergencies, power system operators often require certain wind farms to operate in de-loaded modes. By leveraging the fast response capability of wind farms, the reserved power in deloaded modes can significantly enhance the stability and reliability of power grids. This paper presents a novel methodology that incorporates wind turbines into reinforcement learning-based solutions for frequency response. The proposed approach employs the state-of-the-art reinforcement learning algorithm, surrogategradient- based evolution strategy (GSES), for continuous control of the wind farm output. Our methodology is tested on a modified IEEE-39 bus system, and simulation outcomes demonstrate that the proposed approach can reliably support the frequency of the power system and prevent unnecessary load shedding.

Gao, Wei↗
DOE OSTI2024

Energy performance evaluation of the ASHRAE Guideline 36 control and reinforcement learning–based control using field measurements

This study evaluates the energy performance of ASHRAE Guideline 36–compliant control (ASHRAE 36 control) and reinforcement learning (RL)–based control through experimental field tests and a simulation study. Three field tests were conducted at Oak Ridge National Laboratory’s commercial building test facility in Oak Ridge, Tennessee: a baseline with a baseline conventional control, a test with ASHRAE 36 control, and a test with RL-based control. The selected ASHRAE 36 controls were trim and respond control, as well as variable air volume (VAV) box control. We compared the measured supply air temperature of the rooftop unit, VAV box supply air temperature, and VAV box supply airflow rate across the three test cases. The field data indicated that ASHRAE 36 controls operated as specified by ASHRAE Guideline 36. Based on these data, ASHRAE 36 control achieved a 45 % reduction in hourly averaged HVAC energy consumption compared with the baseline, and RL-based control achieved a 66 % reduction. These potential annual energy savings were confirmed using a calibrated whole-building energy model. Compared with the baseline, ASHRAE 36 control reduced HVAC energy consumption by 42 %, and RL-based control achieved a 54 % reduction. Furthermore, RL-based control reduced total HVAC energy consumption by 21 % more than ASHRAE 36 control.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Reinforcement learning building control approach harnessing imitation learning

Reinforcement learning (RL) has shown significant success in sequential decision making in fields like autonomous vehicles, robotics, marketing and gaming industries. This success has attracted the attention to the RL control approach for building energy systems which are becoming complicated due to the need to optimize for multiple, potentially conflicting, goals like occupant comfort, energy use and grid interactivity. However, for real world applications, RL has several drawbacks like requiring large training data and time, and unstable control behavior during the early exploration process making it infeasible for an application directly to building control tasks. To address these issues, an imitation learning approach is utilized herein where the RL agents starts with a policy transferred from accepted rule based policies and heuristic policies. This approach is successful in reducing the training time, preventing the unstable early exploration behavior and improving upon an accepted rule-based policy - all of these make RL a more practical control approach for real world applications in the domain of building controls.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

A Distributed Reinforcement Learning Yaw Control Approach for Wind Farm Energy Capture Maximization: Preprint

In this paper, we present a reinforcement-learning based distributed approach to wind farm energy capture maximization using yaw control, also known as wake steering. In order to maximize the power output of a wind farm, it is often necessary for individual turbines to decrease their own power output through yaw misalignment so as to deflect their wakes away from downstream turbines. Although using model-based methods to achieve yaw misalignment is one option, a model-free method might be better suited to incorporate factors that are difficult to model or changing conditions. We propose an algorithm that adapts concepts of temporal difference reinforcement learning distributed to a multi-agent environment that allows individual turbines to act so as to optimize overall wind farm output and react to unforeseen disturbances.

controls↗