SEARCH · Search NASA

Results for “Reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 793 records · Page 44

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Replace Human Intelligence with Fast and Smart Geometric Reasoning and Graph Neural Network to Accelerate Next Gen ModSim Workflows

We present an agent-guided approach to CAD geometry decomposition that automates hex/hybrid meshing with graph neural networks (GNNs) to accelerate next-generation ModSim workflows. Our end-to-end pipeline (i) reduces 3D boundary-representation (B-Rep) models to a 2D chordal axis skeleton (CAT) and then to a 1D bipartite graph of surface and curve nodes, (ii) assigns per node labels as Cubit® WebCut actions, (iii) trains a multi-action GNN under supervised learning, and (iv) predicts five surface-node and three curve-node actions on out-of-distribution test geometries. Each graph node carries geometric, topological, and meshing attributes drawn from the B-Rep “skin” and CAT “skeleton,” with two-way mappings across 3D↔2D↔1D representations to maintain traceability back to 3D CAD. The supervised learning model exhibits stable convergence of the binary cross-entropy loss and achieves 98.7% accuracy on unseen lattice models. To operationalize decision-making, we rank predicted commands by geometric significance and prototyped the agent-guided workflow through the Cubit® Meshing PowerTool GUI. As a stretch goal, we explore reinforcement learning (RL) to reduce or remove label requirements and to learn policies for action sequences that maximize total reward (e.g., size of hex-meshable regions and resulting hex mesh quality). When all-hex meshing is not feasible, the agent assists in producing hybrid meshes—prioritizing hex in critical regions and transitioning to tetrahedral elements (tets) elsewhere—maintaining fidelity while ensuring robustness. The overarching objective is to replace manual, heuristics-based decomposition with data-driven, reproducible automation, cutting meshing turnaround time by orders of magnitude. We anticipate direct impact on simulation workflows through intelligent, scalable decomposition of complex CAD models into hex-meshable subdomains.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Closing the Loop between In Situ Stress Complexity and EGS Fracture Complexity

42 ENGINEERING↗

NASA NTRSDate not supplied

Applying Machine Learning Tools for Runway Configuration Decision Support

Determining optimal runway configurations at airports, a responsibility assigned to air traffic controllers, is a challenging task. The decision-making process is intricate and involves consideration of many factors such as prevailing wind condition, convective weather, visibility, cloud ceilings, departure and arrival demand, traffic flow, equipment status, and other airport constraints. In a previous work, we developed a Runway Configuration Assistance tool using an offline reinforcement learning method called conservative Q-learning. In this paper, we evaluate and validate our Runway Configuration Assistance tool as a decision support for air traffic controllers. We validated our tool using three airports with differing levels of complexity: Charlotte Douglas International Airport, Denver International Airport, and Dallas Fort Worth International Airport. We quantified the performance of the Runway Configuration Assistance tool based on (1) agreement with historical air traffic controller decisions and (2) violation of decisions that would be obvious to subject-matter experts. Our tool showed promising results in both performance metrics for the three airports, despite the complexities in the runway configuration decision-making process. We also discuss challenges in using machine learning in general to aid air traffic management and identify deployment considerations for the Runway Configuration Assistance tool.

Runway Configuration Management↗

NASA NTRSDate not supplied

Machine Learning Tools for Runway Configuration Decision Support

Runway Configuration Management↗

NASA NTRS2006

Agent Reward Shaping for Alleviating Traffic Congestion

Traffic congestion problems provide a unique environment to study how multi-agent systems promote desired system level behavior. What is particularly interesting in this class of problems is that no individual action is intrinsically "bad" for the system but that combinations of actions among agents lead to undesirable outcomes, As a consequence, agents need to learn how to coordinate their actions with those of other agents, rather than learn a particular set of "good" actions. This problem is ubiquitous in various traffic problems, including selecting departure times for commuters, routes for airlines, and paths for data routers. In this paper we present a multi-agent approach to two traffic problems, where far each driver, an agent selects the most suitable action using reinforcement learning. The agent rewards are based on concepts from collectives and aim to provide the agents with rewards that are both easy to learn and that if learned, lead to good system level behavior. In the first problem, we study how agents learn the best departure times of drivers in a daily commuting environment and how following those departure times alleviates congestion. In the second problem, we study how agents learn to select desirable routes to improve traffic flow and minimize delays for. all drivers.. In both sets of experiments,. agents using collective-based rewards produced near optimal performance (93-96% of optimal) whereas agents using system rewards (63-68%) barely outperformed random action selection (62-64%) and agents using local rewards (48-72%) performed worse than random in some instances.

Tumer, Kagan↗

NASA NTRS1993

Design issues for a reinforcement-based self-learning fuzzy controller

Fuzzy logic controllers have some often cited advantages over conventional techniques such as PID control: easy implementation, its accommodation to natural language, the ability to cover wider range of operating conditions and others. One major obstacle that hinders its broader application is the lack of a systematic way to develop and modify its rules and as result the creation and modification of fuzzy rules often depends on try-error or pure experimentation. One of the proposed approaches to address this issue is self-learning fuzzy logic controllers (SFLC) that use reinforcement learning techniques to learn the desirability of states and to adjust the consequent part of fuzzy control rules accordingly. Due to the different dynamics of the controlled processes, the performance of self-learning fuzzy controller is highly contingent on the design. The design issue has not received sufficient attention. The issues related to the design of a SFLC for the application to chemical process are discussed and its performance is compared with that of PID and self-tuning fuzzy logic controller.

Yen, John↗

NASA NTRS1992

Design issues of a reinforcement-based self-learning fuzzy controller for petrochemical process control

Fuzzy logic controllers have some often-cited advantages over conventional techniques such as PID control, including easier implementation, accommodation to natural language, and the ability to cover a wider range of operating conditions. One major obstacle that hinders the broader application of fuzzy logic controllers is the lack of a systematic way to develop and modify their rules; as a result the creation and modification of fuzzy rules often depends on trial and error or pure experimentation. One of the proposed approaches to address this issue is a self-learning fuzzy logic controller (SFLC) that uses reinforcement learning techniques to learn the desirability of states and to adjust the consequent part of its fuzzy control rules accordingly. Due to the different dynamics of the controlled processes, the performance of a self-learning fuzzy controller is highly contingent on its design. The design issue has not received sufficient attention. The issues related to the design of a SFLC for application to a petrochemical process are discussed, and its performance is compared with that of a PID and a self-tuning fuzzy logic controller.

Yen, John↗

NASA NTRS2022

SatNet: A Benchmark for Satellite Scheduling Optimization

Satellites provide essential services such as networking and weather tracking, and the number of near-earth and deep space satellites are expected to grow rapidly in the coming years. Communications with terrestrial ground stations is one of the critical functionalities of any space mission. Satellite scheduling is a problem that has been scientifically investigated since the 1970s. A central aspect of this problem is the need to consider resource contention and satellite visibility constraints as they require line of sight. Due to the combinatorial nature of the problem, prior solutions such as linear programs and evolutionary algorithms require extensive compute capabilities to output a feasible schedule for each scenario. Machine learning based scheduling can provide an alternative solution by training a model with historical data and generating a schedule quickly with model inference. We present SatNet, a benchmark for satellite scheduling optimization based on historical data from the NASA Deep Space Network. We propose formulation of the satellite scheduling problem as a Markov Decision Process and use reinforcement learning (RL) policies to generate schedules. The nature of constraints imposed by SatNet differ from other combinatorial optimization problems such as vehicle routing studied in prior literature. Our initial results indicate that RL is an alternative optimization approach that can generate candidate solutions of comparable quality to existing state-of-the-practice results. However, we also find that RL policies overfit to the training dataset and do not generalize well to new data, thereby necessitating continued research on reusable and generalizable agents.

Wilson, Brian↗

DOE OSTI2022

GOOML - Finding Optimization Opportunities for Geothermal Operations: Preprint

Geothermal Operational Optimization with Machine Learning (GOOML) is a transferable and extensible component-based geothermal asset modeling framework that considers complex steamfield relationships and identifies optimization prospects using a data-driven approach. We have used this framework to develop digital twins that provide steamfield operators with an operational environment to analyze and understand historical and forecasted power production, explore new steamfield configuration possibilities, and seek optimal asset management for real world applications. The GOOML modeling software is built on a generic component-based systems framework that allows for both historical and forecast analysis. A GOOML model can perform historical data-assimilation using first-principal thermodynamics to create a meaningful data model. Historical production data can then be coupled with a forecast framework to train machine-learning models of steamfield components to predict future outputs. This modeling environment enables digital exploration of steamfield design configurations and operational scenarios. GOOML digital twins have been developed for steamfields in New Zealand and the United States representing differing power generation and field conditions. These digital twins have been validated by comparing hindcast predictions against historical production data. Reinforcement learning experiments were conducted to demonstrate the ability to programmatically explore the operations space using machine learning agents. Our initial results are compelling; two to five percent increases in annual energy production were demonstrated by the GOOML models with no additional infrastructure build required. GOOML offers a new approach to geothermal operations by applying state-of-the-art machine learning algorithms, comprehensive data analytics, and interaction with digital twins. Through application of these tools, operators will realize greater availability and higher net generation which will increase the cost effectiveness of geothermal energy projects.

access↗

DOE OSTI2022

Multi-objective goal-directed optimization of de novo stable organic radicals for aqueous redox flow batteries

Abstract Advances in the field of goal-directed molecular optimization offer the promise of finding feasible candidates for even the most challenging molecular design applications. One example of a fundamental design challenge is the search for novel stable radical scaffolds for an aqueous redox flow battery that simultaneously satisfy redox requirements at the anode and cathode, as relatively few stable organic radicals are known to exist. To meet this challenge, we develop a new open-source molecular optimization framework based on AlphaZero coupled with a fast, machine-learning-derived surrogate objective trained with nearly 100,000 quantum chemistry simulations. The objective function comprises two graph neural networks: one that predicts adiabatic oxidation and reduction potentials and a second that predicts electron density and local three-dimensional environment, previously shown to be correlated with radical persistence and stability. With no hard-coded knowledge of organic chemistry, the reinforcement learning agent finds molecule candidates that satisfy a precise combination of redox, stability and synthesizability requirements defined at the quantum chemistry level, many of which have reasonable predicted retrosynthetic pathways. The optimized molecules show that alternative stable radical scaffolds may offer a unique profile of stability and redox potentials to enable low-cost symmetric aqueous redox flow batteries.

25 ENERGY STORAGE↗

DOE OSTI2020

Do optimization methods in deep learning applications matter?

With advances in deep learning, exponential data growth and increasing model complexity, developing efficient optimization methods are attracting much research attention. Several implementations favor the use of Conjugate Gradient (CG) and Stochastic Gradient Descent (SGD) as being practical and elegant solutions to achieve quick convergence, however, these optimization processes also present many limitations in learning across deep learning applications. Recent research is exploring higher-order optimization functions as better approaches, but these present very complex computational challenges for practical use. Comparing first and higher-order optimization functions, in this paper, our experiments reveal that Levemberg-Marquardt (LM) significantly supersedes optimal convergence but suffers from very large processing time increasing the training complexity of both, classification and reinforcement learning problems. Our experiments compare off-the-shelf optimization functions(CG, SGD, LM and L-BFGS) in standard CIFAR, MNIST, CartPole and FlappyBird experiments.The paper presents arguments on which optimization functions to use and further, which functions would benefit from parallelization efforts to improve pretraining time and learning rate convergence.

Ozyildirim, Buse Melis↗

DOE OSTI2024

Leverage modern artificial intelligence (AI) enabled systems for waste reduction

Manufacturing industries continue to face challenges in reducing waste, as upstream strategies such as source reduction and product redesign require a deeper understanding of processes compared to conventional recycling methods. Recent advancements in artificial intelligence (AI) and machine learning (ML) have opened new opportunities to integrate modern computational techniques with traditional waste minimization strategies. This paper explores AI-enabled approaches for product redesign, source reduction, and recycling that can significantly reduce waste generation while improving efficiency and sustainability. AI-driven material substitution and lightweighting in product design enable discovery of novel materials with optimized properties, reducing waste without compromising performance. Reinforcement learning models optimize process parameters, raw material specifications, and machine sequencing to minimize production losses, while Industrial Internet of Things (IIoT) systems paired with AI analytics enhance real-time waste tracking, predictive maintenance, and quality inspection. Furthermore, AI-based demand forecasting and production planning reduce overproduction and excess inventory, as demonstrated in industrial applications. In recycling, ML-powered pattern recognition and robotic sorting technologies achieve higher accuracy in waste segregation, directly improving recycling efficiency. Complementary solutions such as smart bins and AI-enabled waste pickup scheduling optimize collection logistics, reducing both costs and emissions. Although implementation requires upfront investment in infrastructure and training, the long-term benefits include higher material efficiency, reduced waste, improved product quality, and stronger sustainability outcomes across the supply chain. By leveraging AI-enabled systems, manufacturers can align waste minimization efforts with circular economy principles, creating scalable solutions for both industry and society.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

Optimal Control of Differentially Private EV Charging: A Scalable Learning Approach Under Uncertainty

Internet of Things (IoT)-enabled electric vehicles (IoEVs) enable intelligent charging coordination that accounts for grid congestion. However, increased data exchange raises privacy concerns, as charging patterns can reveal sensitive driver behavior to grid operators. Here, we propose a differentially private (DP) EV charging framework that enables coordinated control while protecting driver data with theoretical privacy guarantees. Nevertheless, integrating DP inevitably introduces uncertainty into the control strategy for EVs, which can lead to infeasible solutions. To tackle this challenge, we develop a feasible and scalable control algorithm based on constrained reinforcement learning (CRL) and convex hulls. While our framework is designed to handle the uncertainty introduced by DP, it is general and also applicable to other sources of uncertainty in EV charging, such as the stochastic nature of driver behavior and renewable variability. This ensures feasible and privacy-preserving coordination of EV charging at scale. Our method constructs convex hulls within the action space to guarantee feasibility under stochastic constraints and incorporates constraint reduction techniques to improve scalability. Case studies based on IEEE benchmark systems demonstrate that the proposed approach effectively balances feasibility under uncertainty, scalability, and privacy in large-scale EV charging control.

Engineering - Power transmission and distribution↗

DOE OSTI2024

Third-integer Resonant Extraction Regulation System for Mu2e

A third-integer resonant slow extraction system is being developed for Fermilab's Delivery Ring to deliver protons to the upcoming Mu2e experiment. The timescale of the extraction (or spill) duration is 43 milliseconds, which is extremely short and unprecedented. Additionally, the experiment's strict and challenging requirements on the quality of the spill at this time scale has led to the development of a new Spill Regulation System (SRS) design. The SRS primarily consists of three components - slow regulation, fast regulation, and harmonic content suppressor. Contributions to the first two components of the SRS, i.e., Slow Regulation and Fast Regulation subsystems, will be presented in which new adaptive learning algorithm schemes for the slow regulation of the spill -- validated using particle tracking simulations -- shall be described. In addition to these novel methods for the enhancement of the spill regulation system, results of employing Machine Learning in enhancing the performance of the resonant extraction are also presented. At the forefront of applying ML techniques to solve non-linear accelerator control problems, this work includes optimizing the PID gains as well as the replacement of the traditional PID controller using Recurrent Neural Networks and Gated Recurrent Unit (GRU) ML models to achieve efficiencies greater than a PID controller. Cutting-edge on-going Reinforcement Learning efforts, including an actor-critic family of learning algorithms, to regulate the spill rate will be reviewed, as well as present analytical calculations pertaining the transit time of particles in a third-integer resonant extraction. Detailed numerical investigations and validations of such calculations, the model of which could be exported and reliably used in future analytical modeling of any resonant extraction, are discussed.

43 PARTICLE ACCELERATORS↗

DOE OSTI2021

Geothermal Operational Optimization with Machine Learning

The Geothermal Operational Optimization with Machine Learning (GOOML) project has developed a generic and extensible component-based system modeling framework to study complex geothermal fields using a data-driven approach. Through building a digital twin of a geothermal steam field with the GOOML modeling framework, operators can analyze historical and forecasted power production, explore possible steam field configurations, and optimize real world operations, all in a cost-effective digital environment. The GOOML modeling software is based on a historical data-assimilation framework that uses first-principal thermodynamics to model steam field components using historical data, and a forecast framework that uses machine-learning-driven models of steam field components to predict future operations. This modeling framework creates countless new opportunities for digital exploration of steam field design and operations. To date, digital twins have been developed for several steam fields in New Zealand and the United States. These digital twins have been validated by comparing hindcast predictions against historical production data. Field design and operations have been explored using genetic optimization and reinforcement learning. Initial results show compelling and often surprising opportunities for improved design and operation of fields with 2 to 5 percent improvements in annual energy production. GOOML is driving a step-change in geothermal operations by applying state-of-the-art machine learning algorithms, comprehensive data analytics, and a first-of-its-kind intelligent geothermal systems model.

40 EE - Geothermal Technologies Office (EE-4G)↗

NASA NTRS2024

Harnessing Collaborative Learning Automata to Guide Multi-objective Optimization based Inverse Analysis for Structural Damage Identification

Structural damage identification based on physical models is often transformed into an optimization problem that minimizes the difference between measurement information of structure being monitored and the model prediction in the parametric space. However, the objective function in this context often exhibits multimodality, involving high-dimensional variables due to the reliance on finite element models for damage identification. These features pose challenges to optimization algorithms, where entrapment in local solutions can lead to false positives and false negatives in damage identification. In this research, we propose a reinforcement learning based multi-swarm optimizer to tackle such challenges in pursuit of a small yet diverse solution set that can capture the true damage scenario as one of the solutions. The proposed method leverages the flexibility of the particle swarm optimizer and incorporates novel strategies of metaheuristics to realize targeted improvement. To enable the particle swarm to adaptively select the appropriate search strategy based on the current environment, we adopt the learning automata technique, which sidesteps the need for reward strategy selection that is usually ad hoc at each step of the search. The integration harnesses the automatic learning and self-adaptation capabilities of learning automata, enabling the particles to navigate based on environmental signals. This leads to accumulated probabilities tied to advantageous movements, fostering an adaptive exploration of particles in the search space. The proposed approach is first validated through implementing into benchmark test cases with comparisons. It is then applied to structural damage identification with piezoelectric admittance experimental signals. `The results highlight the capability of the algorithm to identify a small solution set with high accuracy to match the actual damage scenario.

Yang Zhang↗

DOE OSTI2022

A resilient network recovery framework against cascading failures with deep graph learning

Because of the increasing importance and dependencies of infrastructure networks and the potential for massive cascading failures in real-world network systems, maintenance optimization to effectively reduce system performance loss caused by diverse disruptions is of significant interest among researchers and practitioners. In this work, a new recovery framework was developed to rapidly identify important system components for maintenance to improve network resilience against cascading failures. Here this work provides distinct advantages to determine an optimal maintenance priority by combining real-time network structure importance with other maintenance prioritization based on customer preference. This approach adopts structural graph embedding and deep reinforcement learning to extract real-time network topology information (such as minimum vertex cover) to update the maintenance priority during the recovery process. Based on the case studies on synthetic networks and a US airport network, the proposed recovery framework with real-time network topology awareness shows better performance than other maintenance prioritization strategies regarding resilience enhancement. This work improves the understanding of how the changing network structure influences maintenance effects. It also provides insights of the practical usefulness of advanced deep learning on helping optimal maintenance prioritization to effectively reduce the intensity and extent of cascading failures.

42 ENGINEERING↗

NASA NTRS1995

Locomotion training of legged robots using hybrid machine learning techniques

In this study artificial neural networks and fuzzy logic are used to control the jumping behavior of a three-link uniped robot. The biped locomotion control problem is an increment of the uniped locomotion control. Study of legged locomotion dynamics indicates that a hierarchical controller is required to control the behavior of a legged robot. A structured control strategy is suggested which includes navigator, motion planner, biped coordinator and uniped controllers. A three-link uniped robot simulation is developed to be used as the plant. Neurocontrollers were trained both online and offline. In the case of on-line training, a reinforcement learning technique was used to train the neurocontroller to make the robot jump to a specified height. After several hundred iterations of training, the plant output achieved an accuracy of 7.4%. However, when jump distance and body angular momentum were also included in the control objectives, training time became impractically long. In the case of off-line training, a three-layered backpropagation (BP) network was first used with three inputs, three outputs and 15 to 40 hidden nodes. Pre-generated data were presented to the network with a learning rate as low as 0.003 in order to reach convergence. The low learning rate required for convergence resulted in a very slow training process which took weeks to learn 460 examples. After training, performance of the neurocontroller was rather poor. Consequently, the BP network was replaced by a Cerebeller Model Articulation Controller (CMAC) network. Subsequent experiments described in this document show that the CMAC network is more suitable to the solution of uniped locomotion control problems in terms of both learning efficiency and performance. A new approach is introduced in this report, viz., a self-organizing multiagent cerebeller model for fuzzy-neural control of uniped locomotion is suggested to improve training efficiency. This is currently being evaluated for a possible patent by NASA, Johnson Space Center. An alternative modular approach is also developed which uses separate controllers for each stage of the running stride. A self-organizing fuzzy-neural controller controls the height, distance and angular momentum of the stride. A CMAC-based controller controls the movement of the leg from the time the foot leaves the ground to the time of landing. Because the leg joints are controlled at each time step during flight, movement is smooth and obstacles can be avoided. Initial results indicate that this approach can yield fast, accurate results.

Simon, William E.↗