Search NASA⌕ Search

SEARCH · Search NASA

Results for “Reinforcement Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
NASA NTRS1991

Approximate reasoning-based learning and control for proximity operations and docking in space

A recently proposed hybrid-neutral-network and fuzzy-logic-control architecture is applied to a fuzzy logic controller developed for attitude control of the Space Shuttle. A model using reinforcement learning and learning from past experience for fine-tuning its knowledge base is proposed. Two main components of this approximate reasoning-based intelligent control (ARIC) model - an action-state evaluation network and action selection network are described as well as the Space Shuttle attitude controller. An ARIC model for the controller is presented, and it is noted that the input layer in each network includes three nodes representing the angle error, angle error rate, and bias node. Preliminary results indicate that the controller can hold the pitch rate within its desired deadband and starts to use the jets at about 500 sec in the run.

Berenji, Hamid R.↗
DOE OSTI2025

Learning plasma dynamics and robust rampdown trajectories with predict-first experiments at TCV

The rampdown phase of a tokamak pulse is difficult to simulate and often exacerbates multiple plasma instabilities. To reduce the risk of disrupting operations, we leverage advances in Scientific Machine Learning (SciML) to combine physics with data-driven models, developing a neural state-space model (NSSM) that predicts plasma dynamics during Tokamak à Configuration Variable (TCV) rampdowns. The NSSM efficiently learns dynamics from a modest dataset of 311 pulses with only five pulses in a reactor-relevant high-performance regime. The NSSM is parallelized across uncertainties, and reinforcement learning (RL) is applied to design trajectories that avoid instability limits. High-performance experiments at TCV show statistically significant improvements in relevant metrics. A predict-first experiment, increasing plasma current by 20% from baseline, demonstrates the NSSM’s ability to make small extrapolations. The developed approach paves the way for designing tokamak controls with robustness to considerable uncertainty and demonstrates the relevance of SciML for fusion experiments.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
NASA NTRS2018

Toward Justifiable Trust in Autonomous Systems Incorporating Human Knowledge in Autonomous Systems through Machine Learning

Trust in Autonomous Systems is largely about humans trusting the decisions made by autonomous systems. This trust can be increased through learning from domain experts. In particular, autonomous systems can learn offline from past mission operations before conducting any operations of its own. Additionally, autonomous systems can learn online by obtaining human feedback during operations. We will discuss several classes of machine learning methods and our application of them to autonomous systems. The first class of methods is anomaly detection, which uses operations data to identify examples of anomalous operations. The second class of methods is inverse reinforcement learning, also known as apprenticeship learning, that takes past operations data as input and yields a controller that is able to duplicate the operations described by the data. The third class is active learning, which identifies examples on which the model is most uncertain and requests domain expert feedback.

Oza, Nikunj C.↗
DOE OSTI2025

Machine learning enabled discovery of superhard and ultrahard carbon polymorphs

The demand for multifunctional materials has motivated the move from near-equilibrium materials to metastable i.e. out-of-equilibrium phases that can meet several desired target properties. The search for such metastable phases with exotic properties is non-trivial and often serendipitous. Inverse design approaches based on evolutionary search have been powerful tools, but such traditional searches have focused on identifying primarily stable and metastable materials with the lowest enthalpy. The inverse design of materials, with a focus on a desired property such as, for example, hardness is a challenging task because of the expensive computational cost involved in sampling multiple structures. The recent advances in machine learning have brought new powerful AI techniques to the forefront which can potentially revolutionize the inverse design and discovery of materials, especially metastable phases capable of meeting multifunctionality. Here, in this work, we develop and apply an automated reinforcement learning workflow for inverse design that integrates first principles physics and atomistic simulations with machine learning (ML), and high-performance computing to allow rapid exploration of the superhard and ultrahard metastable phases of Carbon. We demonstrate an automatic machine learning based inverse design workflow to map new undiscovered metastable states ranging from near equilibrium to those far-from-equilibrium that satisfy multiple property objectives, specifically bulk moduli, shear moduli and hardness. We create a comprehensive library of carbon stable and metastable phases with varying hardness and subsequently shortlist 10 top performing candidate carbon structures, including two newly reported phases, based on their hardness and characterize their temperature dependent mechanical properties. A neural network model is built using featurization of allotropes of carbon to predict the quasi-harmonic Gibbs free energies. The Gibbs free energies of the top performing phases are analyzed to get an estimate of the experimental synthesizability of these superhard and ultrahard carbon phases. In general, we show using machine learning based inverse design approaches how hitherto inaccessible metastable states can be identified and potentially synthesized to meet the demand for multifunctional materials.

Balasubramanian, Karthik [Univ. of Illinois, Chica↗
NASA NTRS2004

Learning to Control Advanced Life Support Systems

Advanced life support systems have many interacting processes and limited resources. Controlling and optimizing advanced life support systems presents unique challenges. In particular, advanced life support systems are nonlinear coupled dynamical systems and it is difficult for humans to take all interactions into account to design an effective control strategy. In this project. we developed several reinforcement learning controllers that actively explore the space of possible control strategies, guided by rewards from a user specified long term objective function. We evaluated these controllers using a discrete event simulation of an advanced life support system. This simulation, called BioSim, designed by Nasa scientists David Kortenkamp and Scott Bell has multiple, interacting life support modules including crew, food production, air revitalization, water recovery, solid waste incineration and power. They are implemented in a consumer/producer relationship in which certain modules produce resources that are consumed by other modules. Stores hold resources between modules. Control of this simulation is via adjusting flows of resources between modules and into/out of stores. We developed adaptive algorithms that control the flow of resources in BioSim. Our learning algorithms discovered several ingenious strategies for maximizing mission length by controlling the air and water recycling systems as well as crop planting schedules. By exploiting non-linearities in the overall system dynamics, the learned controllers easily out- performed controllers written by human experts. In sum, we accomplished three goals. We (1) developed foundations for learning models of coupled dynamical systems by active exploration of the state space, (2) developed and tested algorithms that learn to efficiently control air and water recycling processes as well as crop scheduling in Biosim, and (3) developed an understanding of the role machine learning in designing control systems for advanced life support.

Subramanian, Devika↗
DOE OSTI2024

Machine Learning a Simple Interpretable Short-Range Potential for Silica

A wide array of models, spanning from computationally expensive ab initio methods to a spectrum of force-field approaches, have been developed and employed to probe silica polymorphs and understand growth processes and atomic-level dynamical transitions in silica. However, the quest for a model capable of making accurate predictions with high computational efficiency for various silica polymorphs is still ongoing. Recent developments in short-range machine-learned models, such as GAP and NNPScan, have shown promise in providing reasonable descriptions of silica, but their computational cost remains high compared to force fields such as BKS which are based on simple interpretable functional forms. Here, in this study, we build on the recent success of our reinforcement learning (RL) workflow to derive a new set of optimal parameters for a promising short-range BKS-based model proposed by Soules. We use RL to navigate the eight-dimensional parameter space of the Soules potential using an experimental training data set that includes both local and global structural features from approximately 21 experimentally realized silica polymorphs, including high density phases and porous zeolites. We compare the performance of our machine-learned ML-Soules model with other high quality models including our recent machine-learned parametrization of BKS (ML-BKS), a machine-learned potential (GAP), as well as predictions of ab initio calculations with the highly fidelity SCAN functional. The ML-Soules accurately captures the relative energetic ordering of various polymorphs as well as their structural features at a significantly reduced computational expense. The ML-Soules model also reasonably captures the structure, density, and elastic constants of quartz, as well as metastable silica polymorphs. We further discuss the limitations of the Soules functional form and propose potential enhancements, including the incorporation of additional three-body terms and/or the utilization of different short-ranged functional forms to achieve greater accuracy for both global and local features in the modeling of silica while retaining low computational cost.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Replace Human Intelligence with Fast and Smart Geometric Reasoning and Graph Neural Network to Accelerate Next Gen ModSim Workflows

We present an agent-guided approach to CAD geometry decomposition that automates hex/hybrid meshing with graph neural networks (GNNs) to accelerate next-generation ModSim workflows. Our end-to-end pipeline (i) reduces 3D boundary-representation (B-Rep) models to a 2D chordal axis skeleton (CAT) and then to a 1D bipartite graph of surface and curve nodes, (ii) assigns per node labels as Cubit® WebCut actions, (iii) trains a multi-action GNN under supervised learning, and (iv) predicts five surface-node and three curve-node actions on out-of-distribution test geometries. Each graph node carries geometric, topological, and meshing attributes drawn from the B-Rep “skin” and CAT “skeleton,” with two-way mappings across 3D↔2D↔1D representations to maintain traceability back to 3D CAD. The supervised learning model exhibits stable convergence of the binary cross-entropy loss and achieves 98.7% accuracy on unseen lattice models. To operationalize decision-making, we rank predicted commands by geometric significance and prototyped the agent-guided workflow through the Cubit® Meshing PowerTool GUI. As a stretch goal, we explore reinforcement learning (RL) to reduce or remove label requirements and to learn policies for action sequences that maximize total reward (e.g., size of hex-meshable regions and resulting hex mesh quality). When all-hex meshing is not feasible, the agent assists in producing hybrid meshes—prioritizing hex in critical regions and transitioning to tetrahedral elements (tets) elsewhere—maintaining fidelity while ensuring robustness. The overarching objective is to replace manual, heuristics-based decomposition with data-driven, reproducible automation, cutting meshing turnaround time by orders of magnitude. We anticipate direct impact on simulation workflows through intelligent, scalable decomposition of complex CAD models into hex-meshable subdomains.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Closing the Loop between In Situ Stress Complexity and EGS Fracture Complexity

We present an agent-guided approach to CAD geometry decomposition that automates hex/hybrid meshing with graph neural networks (GNNs) to accelerate next-generation ModSim workflows. Our end-to-end pipeline (i) reduces 3D boundary-representation (B-Rep) models to a 2D chordal axis skeleton (CAT) and then to a 1D bipartite graph of surface and curve nodes, (ii) assigns per node labels as Cubit® WebCut actions, (iii) trains a multi-action GNN under supervised learning, and (iv) predicts five surface-node and three curve-node actions on out-of-distribution test geometries. Each graph node carries geometric, topological, and meshing attributes drawn from the B-Rep “skin” and CAT “skeleton,” with two-way mappings across 3D↔2D↔1D representations to maintain traceability back to 3D CAD. The supervised learning model exhibits stable convergence of the binary cross-entropy loss and achieves 98.7% accuracy on unseen lattice models. To operationalize decision-making, we rank predicted commands by geometric significance and prototyped the agent-guided workflow through the Cubit® Meshing PowerTool GUI. As a stretch goal, we explore reinforcement learning (RL) to reduce or remove label requirements and to learn policies for action sequences that maximize total reward (e.g., size of hex-meshable regions and resulting hex mesh quality). When all-hex meshing is not feasible, the agent assists in producing hybrid meshes—prioritizing hex in critical regions and transitioning to tetrahedral elements (tets) elsewhere—maintaining fidelity while ensuring robustness. The overarching objective is to replace manual, heuristics-based decomposition with data-driven, reproducible automation, cutting meshing turnaround time by orders of magnitude. We anticipate direct impact on simulation workflows through intelligent, scalable decomposition of complex CAD models into hex-meshable subdomains.

42 ENGINEERING↗
NASA NTRSDate not supplied

Applying Machine Learning Tools for Runway Configuration Decision Support

Determining optimal runway configurations at airports, a responsibility assigned to air traffic controllers, is a challenging task. The decision-making process is intricate and involves consideration of many factors such as prevailing wind condition, convective weather, visibility, cloud ceilings, departure and arrival demand, traffic flow, equipment status, and other airport constraints. In a previous work, we developed a Runway Configuration Assistance tool using an offline reinforcement learning method called conservative Q-learning. In this paper, we evaluate and validate our Runway Configuration Assistance tool as a decision support for air traffic controllers. We validated our tool using three airports with differing levels of complexity: Charlotte Douglas International Airport, Denver International Airport, and Dallas Fort Worth International Airport. We quantified the performance of the Runway Configuration Assistance tool based on (1) agreement with historical air traffic controller decisions and (2) violation of decisions that would be obvious to subject-matter experts. Our tool showed promising results in both performance metrics for the three airports, despite the complexities in the runway configuration decision-making process. We also discuss challenges in using machine learning in general to aid air traffic management and identify deployment considerations for the Runway Configuration Assistance tool.

Runway Configuration Management↗
NASA NTRSDate not supplied

Machine Learning Tools for Runway Configuration Decision Support

Determining optimal runway configurations at airports, a responsibility assigned to air traffic controllers, is a challenging task. The decision-making process is intricate and involves consideration of many factors such as prevailing wind condition, convective weather, visibility, cloud ceilings, departure and arrival demand, traffic flow, equipment status, and other airport constraints. In a previous work, we developed a Runway Configuration Assistance tool using an offline reinforcement learning method called conservative Q-learning. In this paper, we evaluate and validate our Runway Configuration Assistance tool as a decision support for air traffic controllers. We validated our tool using three airports with differing levels of complexity: Charlotte Douglas International Airport, Denver International Airport, and Dallas Fort Worth International Airport. We quantified the performance of the Runway Configuration Assistance tool based on (1) agreement with historical air traffic controller decisions and (2) violation of decisions that would be obvious to subject-matter experts. Our tool showed promising results in both performance metrics for the three airports, despite the complexities in the runway configuration decision-making process. We also discuss challenges in using machine learning in general to aid air traffic management and identify deployment considerations for the Runway Configuration Assistance tool.

Runway Configuration Management↗
NASA NTRS2006

Agent Reward Shaping for Alleviating Traffic Congestion

Traffic congestion problems provide a unique environment to study how multi-agent systems promote desired system level behavior. What is particularly interesting in this class of problems is that no individual action is intrinsically "bad" for the system but that combinations of actions among agents lead to undesirable outcomes, As a consequence, agents need to learn how to coordinate their actions with those of other agents, rather than learn a particular set of "good" actions. This problem is ubiquitous in various traffic problems, including selecting departure times for commuters, routes for airlines, and paths for data routers. In this paper we present a multi-agent approach to two traffic problems, where far each driver, an agent selects the most suitable action using reinforcement learning. The agent rewards are based on concepts from collectives and aim to provide the agents with rewards that are both easy to learn and that if learned, lead to good system level behavior. In the first problem, we study how agents learn the best departure times of drivers in a daily commuting environment and how following those departure times alleviates congestion. In the second problem, we study how agents learn to select desirable routes to improve traffic flow and minimize delays for. all drivers.. In both sets of experiments,. agents using collective-based rewards produced near optimal performance (93-96% of optimal) whereas agents using system rewards (63-68%) barely outperformed random action selection (62-64%) and agents using local rewards (48-72%) performed worse than random in some instances.

Tumer, Kagan↗
NASA NTRS1993

Design issues for a reinforcement-based self-learning fuzzy controller

Fuzzy logic controllers have some often cited advantages over conventional techniques such as PID control: easy implementation, its accommodation to natural language, the ability to cover wider range of operating conditions and others. One major obstacle that hinders its broader application is the lack of a systematic way to develop and modify its rules and as result the creation and modification of fuzzy rules often depends on try-error or pure experimentation. One of the proposed approaches to address this issue is self-learning fuzzy logic controllers (SFLC) that use reinforcement learning techniques to learn the desirability of states and to adjust the consequent part of fuzzy control rules accordingly. Due to the different dynamics of the controlled processes, the performance of self-learning fuzzy controller is highly contingent on the design. The design issue has not received sufficient attention. The issues related to the design of a SFLC for the application to chemical process are discussed and its performance is compared with that of PID and self-tuning fuzzy logic controller.

Yen, John↗
NASA NTRS1992

Design issues of a reinforcement-based self-learning fuzzy controller for petrochemical process control

Fuzzy logic controllers have some often-cited advantages over conventional techniques such as PID control, including easier implementation, accommodation to natural language, and the ability to cover a wider range of operating conditions. One major obstacle that hinders the broader application of fuzzy logic controllers is the lack of a systematic way to develop and modify their rules; as a result the creation and modification of fuzzy rules often depends on trial and error or pure experimentation. One of the proposed approaches to address this issue is a self-learning fuzzy logic controller (SFLC) that uses reinforcement learning techniques to learn the desirability of states and to adjust the consequent part of its fuzzy control rules accordingly. Due to the different dynamics of the controlled processes, the performance of a self-learning fuzzy controller is highly contingent on its design. The design issue has not received sufficient attention. The issues related to the design of a SFLC for application to a petrochemical process are discussed, and its performance is compared with that of a PID and a self-tuning fuzzy logic controller.

Yen, John↗
NASA NTRS2022

SatNet: A Benchmark for Satellite Scheduling Optimization

Satellites provide essential services such as networking and weather tracking, and the number of near-earth and deep space satellites are expected to grow rapidly in the coming years. Communications with terrestrial ground stations is one of the critical functionalities of any space mission. Satellite scheduling is a problem that has been scientifically investigated since the 1970s. A central aspect of this problem is the need to consider resource contention and satellite visibility constraints as they require line of sight. Due to the combinatorial nature of the problem, prior solutions such as linear programs and evolutionary algorithms require extensive compute capabilities to output a feasible schedule for each scenario. Machine learning based scheduling can provide an alternative solution by training a model with historical data and generating a schedule quickly with model inference. We present SatNet, a benchmark for satellite scheduling optimization based on historical data from the NASA Deep Space Network. We propose formulation of the satellite scheduling problem as a Markov Decision Process and use reinforcement learning (RL) policies to generate schedules. The nature of constraints imposed by SatNet differ from other combinatorial optimization problems such as vehicle routing studied in prior literature. Our initial results indicate that RL is an alternative optimization approach that can generate candidate solutions of comparable quality to existing state-of-the-practice results. However, we also find that RL policies overfit to the training dataset and do not generalize well to new data, thereby necessitating continued research on reusable and generalizable agents.

Wilson, Brian↗
DOE OSTI2024

Leverage modern artificial intelligence (AI) enabled systems for waste reduction

Manufacturing industries continue to face challenges in reducing waste, as upstream strategies such as source reduction and product redesign require a deeper understanding of processes compared to conventional recycling methods. Recent advancements in artificial intelligence (AI) and machine learning (ML) have opened new opportunities to integrate modern computational techniques with traditional waste minimization strategies. This paper explores AI-enabled approaches for product redesign, source reduction, and recycling that can significantly reduce waste generation while improving efficiency and sustainability. AI-driven material substitution and lightweighting in product design enable discovery of novel materials with optimized properties, reducing waste without compromising performance. Reinforcement learning models optimize process parameters, raw material specifications, and machine sequencing to minimize production losses, while Industrial Internet of Things (IIoT) systems paired with AI analytics enhance real-time waste tracking, predictive maintenance, and quality inspection. Furthermore, AI-based demand forecasting and production planning reduce overproduction and excess inventory, as demonstrated in industrial applications. In recycling, ML-powered pattern recognition and robotic sorting technologies achieve higher accuracy in waste segregation, directly improving recycling efficiency. Complementary solutions such as smart bins and AI-enabled waste pickup scheduling optimize collection logistics, reducing both costs and emissions. Although implementation requires upfront investment in infrastructure and training, the long-term benefits include higher material efficiency, reduced waste, improved product quality, and stronger sustainability outcomes across the supply chain. By leveraging AI-enabled systems, manufacturers can align waste minimization efforts with circular economy principles, creating scalable solutions for both industry and society.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Optimal Control of Differentially Private EV Charging: A Scalable Learning Approach Under Uncertainty

Internet of Things (IoT)-enabled electric vehicles (IoEVs) enable intelligent charging coordination that accounts for grid congestion. However, increased data exchange raises privacy concerns, as charging patterns can reveal sensitive driver behavior to grid operators. Here, we propose a differentially private (DP) EV charging framework that enables coordinated control while protecting driver data with theoretical privacy guarantees. Nevertheless, integrating DP inevitably introduces uncertainty into the control strategy for EVs, which can lead to infeasible solutions. To tackle this challenge, we develop a feasible and scalable control algorithm based on constrained reinforcement learning (CRL) and convex hulls. While our framework is designed to handle the uncertainty introduced by DP, it is general and also applicable to other sources of uncertainty in EV charging, such as the stochastic nature of driver behavior and renewable variability. This ensures feasible and privacy-preserving coordination of EV charging at scale. Our method constructs convex hulls within the action space to guarantee feasibility under stochastic constraints and incorporates constraint reduction techniques to improve scalability. Case studies based on IEEE benchmark systems demonstrate that the proposed approach effectively balances feasibility under uncertainty, scalability, and privacy in large-scale EV charging control.

Engineering - Power transmission and distribution↗
DOE OSTI2024

Third-integer Resonant Extraction Regulation System for Mu2e

A third-integer resonant slow extraction system is being developed for Fermilab's Delivery Ring to deliver protons to the upcoming Mu2e experiment. The timescale of the extraction (or spill) duration is 43 milliseconds, which is extremely short and unprecedented. Additionally, the experiment's strict and challenging requirements on the quality of the spill at this time scale has led to the development of a new Spill Regulation System (SRS) design. The SRS primarily consists of three components - slow regulation, fast regulation, and harmonic content suppressor. Contributions to the first two components of the SRS, i.e., Slow Regulation and Fast Regulation subsystems, will be presented in which new adaptive learning algorithm schemes for the slow regulation of the spill -- validated using particle tracking simulations -- shall be described. In addition to these novel methods for the enhancement of the spill regulation system, results of employing Machine Learning in enhancing the performance of the resonant extraction are also presented. At the forefront of applying ML techniques to solve non-linear accelerator control problems, this work includes optimizing the PID gains as well as the replacement of the traditional PID controller using Recurrent Neural Networks and Gated Recurrent Unit (GRU) ML models to achieve efficiencies greater than a PID controller. Cutting-edge on-going Reinforcement Learning efforts, including an actor-critic family of learning algorithms, to regulate the spill rate will be reviewed, as well as present analytical calculations pertaining the transit time of particles in a third-integer resonant extraction. Detailed numerical investigations and validations of such calculations, the model of which could be exported and reliably used in future analytical modeling of any resonant extraction, are discussed.

43 PARTICLE ACCELERATORS↗
NASA NTRS2024

Harnessing Collaborative Learning Automata to Guide Multi-objective Optimization based Inverse Analysis for Structural Damage Identification

Structural damage identification based on physical models is often transformed into an optimization problem that minimizes the difference between measurement information of structure being monitored and the model prediction in the parametric space. However, the objective function in this context often exhibits multimodality, involving high-dimensional variables due to the reliance on finite element models for damage identification. These features pose challenges to optimization algorithms, where entrapment in local solutions can lead to false positives and false negatives in damage identification. In this research, we propose a reinforcement learning based multi-swarm optimizer to tackle such challenges in pursuit of a small yet diverse solution set that can capture the true damage scenario as one of the solutions. The proposed method leverages the flexibility of the particle swarm optimizer and incorporates novel strategies of metaheuristics to realize targeted improvement. To enable the particle swarm to adaptively select the appropriate search strategy based on the current environment, we adopt the learning automata technique, which sidesteps the need for reward strategy selection that is usually ad hoc at each step of the search. The integration harnesses the automatic learning and self-adaptation capabilities of learning automata, enabling the particles to navigate based on environmental signals. This leads to accumulated probabilities tied to advantageous movements, fostering an adaptive exploration of particles in the search space. The proposed approach is first validated through implementing into benchmark test cases with comparisons. It is then applied to structural damage identification with piezoelectric admittance experimental signals. `The results highlight the capability of the algorithm to identify a small solution set with high accuracy to match the actual damage scenario.

Yang Zhang↗