Search NASA⌕ Search

SEARCH · Search NASA

Results for “Offline Reinforcement Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Optimization of Airport Runway Configuration with Forecast-Augmented Offline Reinforcement Learning

Runway configuration Management (RCM) governs the optimal utilization of runways based on variables such as traffic and meteorological conditions, making it a daunting task in air traffic management due to its dependency on volatile operational and environmental factors. This paper improves upon our previous work [1] on using offline model-free reinforcement learning for creating a Runway Configuration Assistance (RCA) decision-support tool. A novel integration of forecast data from LAMP (Localized Aviation Model Output Statistics Program) and TAF (Terminal Area Forecast) is introduced, enhancing the tool’s accuracy and also its adaptability to quick wind changes. The performance is evaluated using two major US airports, Charlotte Douglas International Airport (CLT) and Denver International Airport (DEN). To counter scalability issues presented by the addition of discrete forecast variables, we transitioned to a continuous state space model, ensuring scalability and inclusion of longer forecast data. The results of our experiments reflect significant improvements in the RCA tool’s prediction accuracy.

Sumanth Nethi↗
DOE OSTI2023

Self-Driving Telescopes: Autonomous Scheduling of Astronomical Observation Campaigns with Offline Reinforcement Learning

Modern astronomical experiments are designed to achieve multiple scientific goals, from studies of galaxy evolution to cosmic acceleration. These goals require data of many different classes of night-sky objects, each of which has a particular set of observational needs. These observational needs are typically in strong competition with one another. This poses a challenging multi-objective optimization problem that remains unsolved. The effectiveness of Reinforcement Learning (RL) as a valuable paradigm for training autonomous systems has been well-demonstrated, and it may provide the basis for self-driving telescopes capable of optimizing the scheduling for astronomy campaigns. Simulated datasets containing examples of interactions between a telescope and a discrete set of sky locations on the celestial sphere can be used to train an RL model to sequentially gather data from these several locations to maximize a cumulative reward as a measure of the quality of the data gathered. We use simulated data to test and compare multiple implementations of a Deep Q-Network (DQN) for the task of optimizing the schedule of observations from the Stone Edge Observatory (SEO). We combine multiple improvements on the DQN and adjustments to the dataset, showing that DQNs can achieve an average reward of 87%+-6% of the maximum achievable reward in each state on the test set. This is the first comparison of offline RL algorithms for a particular astronomical challenge and the first open-source framework for performing such a comparison and assessment task.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRSDate not supplied

Runway Configuration Management with Offline Reinforcement Learning

Runway configuration management (RCM) is a challenging task, and it affects the efficiency of the National Airspace System (NAS) and airport surface operations significantly. Each airport, depending on the geometry, capacity, local climate patterns, etc. has multiple configurations for the runway usage for arriving and departing flights. Many factors such as the incoming/outgoing traffic load, wind direction and speed, convective weather, cloud ceiling and other environmental factors might affect the choice of a runway configuration at any point in time. However, other factors such as safety measures and regulations, noise abatement, capacity of each configuration, and preference of the air traffic controllers (ATCs) can also play a significant role in selecting the configuration. A sub-optimal selection of the runway configuration, or delay in making configuration changes might result in significant increase in taxi times for aircraft on the surface of the airport, fuel and energy use of the aircraft, and maintenance costs. It can also lead to safety concerns, such as an aircraft performing one or more go-arounds before being able to land. All these factors make RCM an extremely important and challenging decision-making process for the ATCs. The current state of practice sets the runway configuration by the ATCs based on relevant information available at the time including weather, traffic, noise abatement, safety bounds, etc. This makes the decision-making process subjective based on the accuracy of the available information and the bias in human decision making. Unfortunately, this approach yields poor results (e.g., significant delays) if the predicted outcomes are uncertain and their relative impact is not well understood. This is especially evident when the uncertainty increases the size of possible predicted outcomes (combinatorial explosion in possible scenarios) that cannot be handled by human reasoning. On the other hand, an automated approach based on machine intelligence can make use of historical data and search through all (or significant amount of) possible scenarios under uncertainty and make well-informed decisions.

Milad Memarzadeh↗
DOE OSTI2023

Unlocking Autonomous Telescopes through Reinforcement Learning: An Offline Framework and Insights from a Case Study

Optimizing observational astronomy campaigns is becoming a complex and expensive task for next-generation telescopes, where manual planning of observations may tend to reach suboptimal results in terms of optimization.Reinforcement Learning (RL) has been well-demonstrated as a valuable approach for training autonomous systems, and it may provide the basis for self-driving telescopes capable of scanning the sky and optimizing the scheduling for astronomy campaigns.We have developed a framework for the optimization of telescope scheduling using RL techniques, based on a dataset containing data on a discrete set of sky locations that the telescope should visit, and a reward metric. We compared several RL algorithms applied to an offline simulation dataset based at the Stone Edge Observatory, considering a discrete set of sky locations to visit and using “t-effective” as a reward metric, a measure of the quality of the data.Deep Q-Networks (DQNs), belonging to the class of value-based methods, have shown remarkable success in the optimization of astronomical observations in our dataset. In the full environment, the average reward value in each state was found to be 92%±5% of the maximum possible reward, while on the test set it resulted in 87%±9% of the maximum possible reward.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Efficient and assured reinforcement learning-based building HVAC control with heterogeneous expert-guided training

Abstract Building heating, ventilation, and air conditioning (HVAC) systems account for nearly half of building energy consumption and $$20\%$$ of total energy consumption in the US. Their operation is also crucial for ensuring the physical and mental health of building occupants. Compared with traditional model-based HVAC control methods, the recent model-free deep reinforcement learning (DRL) based methods have shown good performance while do not require the development of detailed and costly physical models. However, these model-free DRL approaches often suffer from long training time to reach a good performance, which is a major obstacle for their practical deployment. In this work, we present a systematic approach to accelerate online reinforcement learning for HVAC control by taking full advantage of the knowledge from domain experts in various forms . Specifically, the algorithm stages include learning expert functions from existing abstract physical models and from historical data via offline reinforcement learning, integrating the expert functions with rule-based guidelines, conducting training guided by the integrated expert function and performing policy initialization from distilled expert function. Moreover, to ensure that the learned DRL-based HVAC controller can effectively keep room temperature within the comfortable range for occupants, we design a runtime shielding framework to reduce the temperature violation rate and incorporate the learned controller into it. Experimental results demonstrate up to 8.8 X speedup in DRL training from our approach over previous methods, with low temperature violation rate.

Xu, Shichao↗
NASA NTRSDate not supplied

Applying Machine Learning Tools for Runway Configuration Decision Support

Determining optimal runway configurations at airports, a responsibility assigned to air traffic controllers, is a challenging task. The decision-making process is intricate and involves consideration of many factors such as prevailing wind condition, convective weather, visibility, cloud ceilings, departure and arrival demand, traffic flow, equipment status, and other airport constraints. In a previous work, we developed a Runway Configuration Assistance tool using an offline reinforcement learning method called conservative Q-learning. In this paper, we evaluate and validate our Runway Configuration Assistance tool as a decision support for air traffic controllers. We validated our tool using three airports with differing levels of complexity: Charlotte Douglas International Airport, Denver International Airport, and Dallas Fort Worth International Airport. We quantified the performance of the Runway Configuration Assistance tool based on (1) agreement with historical air traffic controller decisions and (2) violation of decisions that would be obvious to subject-matter experts. Our tool showed promising results in both performance metrics for the three airports, despite the complexities in the runway configuration decision-making process. We also discuss challenges in using machine learning in general to aid air traffic management and identify deployment considerations for the Runway Configuration Assistance tool.

Runway Configuration Management↗
NASA NTRSDate not supplied

Machine Learning Tools for Runway Configuration Decision Support

Determining optimal runway configurations at airports, a responsibility assigned to air traffic controllers, is a challenging task. The decision-making process is intricate and involves consideration of many factors such as prevailing wind condition, convective weather, visibility, cloud ceilings, departure and arrival demand, traffic flow, equipment status, and other airport constraints. In a previous work, we developed a Runway Configuration Assistance tool using an offline reinforcement learning method called conservative Q-learning. In this paper, we evaluate and validate our Runway Configuration Assistance tool as a decision support for air traffic controllers. We validated our tool using three airports with differing levels of complexity: Charlotte Douglas International Airport, Denver International Airport, and Dallas Fort Worth International Airport. We quantified the performance of the Runway Configuration Assistance tool based on (1) agreement with historical air traffic controller decisions and (2) violation of decisions that would be obvious to subject-matter experts. Our tool showed promising results in both performance metrics for the three airports, despite the complexities in the runway configuration decision-making process. We also discuss challenges in using machine learning in general to aid air traffic management and identify deployment considerations for the Runway Configuration Assistance tool.

Runway Configuration Management↗
NASA NTRSDate not supplied

Airport Runway Configuration Management with Offline Model-free Reinforcement Learning

Runway configuration management (RCM) deals with the optimal selection of runways to operate on (for arrivals and departures) based on traffic, surface wind speed, wind direction and other environmental variables. RCM is one of the most challenging tasks in air traffic management, as it relies on operational and environmental variables (e.g., weather forecast) that are highly uncertain and complex to model. In this paper, an innovative and automated approach is deployed using offline model-free reinforcement learning to provide decision-support for RCM. The proposed technology processes historical data about variables of interest, decisions made regarding RCM, and their subsequent outcome, to identify a policy that would encourage good decisions and avoid the poor ones. The policy search is guided by an appropriately chosen weighted utility function (e.g., based on minimizing delays and go-arounds). Finally, the performance of the proposed tool is validated using Charlotte Douglas International Airport as the case study, which shows that the proposed method is superior to other conventional rule-based approaches.

Milad Memarzadeh↗
NASA NTRSDate not supplied

Airport Runway Configuration Management with Offline Model-free Reinforcement Learning

Runway configuration management (RCM) deals with the optimal selection of runways to operate on (for arrivals and departures) based on traffic, surface wind speed, wind direction and other environmental variables. RCM is one of the most challenging tasks in air traffic management, as it relies on operational and environmental variables (e.g., weather forecast) that are highly uncertain and complex to model. In this paper, an innovative and automated approach is deployed using offline model-free reinforcement learning to provide decision-support for RCM. The proposed technology processes historical data about variables of interest, decisions made regarding RCM, and their subsequent outcome, to identify a policy that would encourage good decisions and avoid the poor ones. The policy search is guided by an appropriately chosen weighted utility function (e.g., based on minimizing delays and go-arounds). Finally, the performance of the proposed tool is validated using Charlotte Douglas International Airport as the case study, which shows that the proposed method is superior to other conventional rule-based approaches.

Milad Memarzadeh↗
DOE OSTI2024

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

Deep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Correlated Trajectory Uncertainty for Adaptive Sequential Decision Making

One of the great challenges with decision making tasks on real world systems is the fact that data is sparse and acquiring additional data is expensive. In these cases, it is often crucial to make a model of the environment to assist in making decisions. At the same time, limited data means that learned models are erroneous, making it just as important to equip the model with good predictive uncertainties. In the context of learning sequential decision making policies, these uncertainties can prove useful for informing which data to collect for the greatest improvement in policy performance \citep{mehta2021experimental, mehta2022exploration} or informing the policy about unsure regions of state and action space to avoid during test time \citep{yu2020mopo}. Additionally, assuming that realistic samples of the environment can be drawn, an adaptable policy can be trained that attempts to make optimal decisions for any given possible instance of the environment \citep{ghosh2022offline, chen2021offline}. In this work, we examine the so-called ``probabilistic neural network'' (PNN) model that is ubiquitous in model-based reinforcement learning (MBRL) works. We argue that while PNN models may have good marginal uncertainties, they form a distribution of non-smooth transition functions. Not only are these samples unrealistic and may hamper adaptability, but we also assert that this leads to poor uncertainty estimates when predicting multiple step trajectory estimates. To address this issue, we propose a simple sampling method that can be implemented on top of pre-existing models.We evaluate our sampling technique on a number of environments, including a realistic nuclear fusion task, and find that, not only do smooth transition function samples produce more calibrated uncertainties, but they also lead to better downstream performance for an adaptive policy.

Offline Reinforcement Learning↗
NASA NTRSDate not supplied

Collision Avoidance Approach Using Deep Reinforcement Learning

A method to enable autonomous robots moving in a 2D space collision free motivates the purposed approach for collision avoidance for autonomous UAM vehicles. Challenges of autonomous collision free navigation for both problems are similar. Agents in each environment do not know the intent, or goal, of the other. Finding the time efficient paths require some level of anticipation with neighboring agents which is computationally expensive. In the original work, these obstacles were overcome with a novel application of deep reinforcement learning which offloads the online computation to an offline learning algorithm. A value network that encodes the estimated time to the goal given the agent’s state and the observable portion of the other agent’s state is trained on a baseline policy and further refined with reinforcement learning to promote time efficient collision free navigation. Online, the value network efficiently informs the agent’s decision making in the face of uncertainty of the other agent’s next move. In this paper, challenges extending this methodology to the 3D environment of autonomous UAM vehicles with kinematic constraints are discussed and initial results shown.

Collision Avoidance↗
NASA NTRSDate not supplied

Collision Avoidance Approach Using Deep Reinforcement Learning

A method to enable autonomous robots moving in a 2D space collision free motivates the purposed approach for collision avoidance for autonomous UAM vehicles. Challenges of autonomous collision free navigation for both problems are similar. Agents in each environment do not know the intent, or goal, of the other. Finding the time efficient paths require some level of anticipation with neighboring agents which is computationally expensive. In the original work, these obstacles were overcome with a novel application of deep reinforcement learning which offloads the online computation to an offline learning algorithm. A value network that encodes the estimated time to the goal given the agent’s state and the observable portion of the other agent’s state is trained on a baseline policy and further refined with reinforcement learning to promote time efficient collision free navigation. Online, the value network efficiently informs the agent’s decision making in the face of uncertainty of the other agent’s next move. In this paper, challenges extending this methodology to the 3D environment of autonomous UAM vehicles with kinematic constraints are discussed and initial results shown.

Collision Avoidance↗
NASA NTRS1995

Locomotion training of legged robots using hybrid machine learning techniques

In this study artificial neural networks and fuzzy logic are used to control the jumping behavior of a three-link uniped robot. The biped locomotion control problem is an increment of the uniped locomotion control. Study of legged locomotion dynamics indicates that a hierarchical controller is required to control the behavior of a legged robot. A structured control strategy is suggested which includes navigator, motion planner, biped coordinator and uniped controllers. A three-link uniped robot simulation is developed to be used as the plant. Neurocontrollers were trained both online and offline. In the case of on-line training, a reinforcement learning technique was used to train the neurocontroller to make the robot jump to a specified height. After several hundred iterations of training, the plant output achieved an accuracy of 7.4%. However, when jump distance and body angular momentum were also included in the control objectives, training time became impractically long. In the case of off-line training, a three-layered backpropagation (BP) network was first used with three inputs, three outputs and 15 to 40 hidden nodes. Pre-generated data were presented to the network with a learning rate as low as 0.003 in order to reach convergence. The low learning rate required for convergence resulted in a very slow training process which took weeks to learn 460 examples. After training, performance of the neurocontroller was rather poor. Consequently, the BP network was replaced by a Cerebeller Model Articulation Controller (CMAC) network. Subsequent experiments described in this document show that the CMAC network is more suitable to the solution of uniped locomotion control problems in terms of both learning efficiency and performance. A new approach is introduced in this report, viz., a self-organizing multiagent cerebeller model for fuzzy-neural control of uniped locomotion is suggested to improve training efficiency. This is currently being evaluated for a possible patent by NASA, Johnson Space Center. An alternative modular approach is also developed which uses separate controllers for each stage of the running stride. A self-organizing fuzzy-neural controller controls the height, distance and angular momentum of the stride. A CMAC-based controller controls the movement of the leg from the time the foot leaves the ground to the time of landing. Because the leg joints are controlled at each time step during flight, movement is smooth and obstacles can be avoided. Initial results indicate that this approach can yield fast, accurate results.

Simon, William E.↗
NASA NTRSDate not supplied

An Online Reinforcement Learning Controller Design For Mars Ascent Vehicle

This paper presents a neural network (NN) approximator-based online reinforcement learning (ORL) controller design for Mars Ascent Vehicle (MAV) under parametric variation and significant external disturbances. The ORL controller, which does not require any offline training, involves two NNs where an action NN produces optimal short-term control performance while a critic NN evaluates the performance of the action NN using an approximated cost function. The simulation example with comparisons against baseline Proportional-Integral-Derivative (PID) and gain scheduled pole-placement PID (GS-PP-PID) controllers show the proposed controller’s effectiveness and robustness under parametric variation and high external disturbances.

Han Woong Bae↗
DOE OSTI2025

Safe and Robust Binary Classification and Fault Detection Using Reinforcement Learning

In this paper, we propose a learning-based method utilizing the Soft Actor-Critic (SAC) algorithm to train a binary Support Vector Machine (SVM) classifier. This classifier is designed to identify valid input spaces in high-dimensional, highly constrained systems while minimizing the total runtime of offline simulations. The simulations adapt their runtime based on the likelihood that a given training input will be informative to the classifier. Furthermore, we introduce a method for using the trained SAC model to predict whether a desired system input is likely to violate constraints, along with a technique to adjust the input as necessary. Additionally, we explore the potential of this model to detect faults or adversarial attacks within the system. The effectiveness of our approach is demonstrated through various simulations of challenging classification problems and a constrained quadrotor model.

Netter, Josh [Georgia Institute of Technology, Atl↗
DOE OSTI2024

Learning Robust Marking Policies for Adaptive Mesh Refinement

Here in this work, we revisit the marking decisions made in the standard adaptive finite element method (AFEM). Experience shows that a naïve marking policy leads to inefficient use of computational resources for adaptive mesh refinement (AMR). Consequently, using AMR in practice often involves ad-hoc or time-consuming offline parameter tuning to set appropriate parameters for the marking subroutine. To address these practical concerns, we recast AMR as a Markov decision process in which refinement parameters can be selected on-the-fly at run time, without the need for pre-tuning by expert users. In this new paradigm, the refinement parameters are also chosen adaptively via a marking policy that can be optimized using methods from reinforcement learning. We use the Poisson equation to demonstrate our techniques on h- and hp-refinement benchmark problems, and our experiments suggest that superior marking policies remain undiscovered for many classical AFEM applications. Furthermore, an unexpected observation from this work is that marking policies trained on one family of PDEs are sometimes robust enough to perform well on problems far outside the training family. For illustration, we show that a simple hp-refinement policy trained on 2D domains with only a single re-entrant corner can be deployed on far more complicated 2D domains, and even 3D domains, without significant performance loss. For reproduction and broader adoption, we accompany this work with an open-source implementation of our methods.

97 MATHEMATICS AND COMPUTING↗