SEARCH · Search NASA

Results for “reinforcement learning (RL)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Replace Human Intelligence with Fast and Smart Geometric Reasoning and Graph Neural Network to Accelerate Next Gen ModSim Workflows

We present an agent-guided approach to CAD geometry decomposition that automates hex/hybrid meshing with graph neural networks (GNNs) to accelerate next-generation ModSim workflows. Our end-to-end pipeline (i) reduces 3D boundary-representation (B-Rep) models to a 2D chordal axis skeleton (CAT) and then to a 1D bipartite graph of surface and curve nodes, (ii) assigns per node labels as Cubit® WebCut actions, (iii) trains a multi-action GNN under supervised learning, and (iv) predicts five surface-node and three curve-node actions on out-of-distribution test geometries. Each graph node carries geometric, topological, and meshing attributes drawn from the B-Rep “skin” and CAT “skeleton,” with two-way mappings across 3D↔2D↔1D representations to maintain traceability back to 3D CAD. The supervised learning model exhibits stable convergence of the binary cross-entropy loss and achieves 98.7% accuracy on unseen lattice models. To operationalize decision-making, we rank predicted commands by geometric significance and prototyped the agent-guided workflow through the Cubit® Meshing PowerTool GUI. As a stretch goal, we explore reinforcement learning (RL) to reduce or remove label requirements and to learn policies for action sequences that maximize total reward (e.g., size of hex-meshable regions and resulting hex mesh quality). When all-hex meshing is not feasible, the agent assists in producing hybrid meshes—prioritizing hex in critical regions and transitioning to tetrahedral elements (tets) elsewhere—maintaining fidelity while ensuring robustness. The overarching objective is to replace manual, heuristics-based decomposition with data-driven, reproducible automation, cutting meshing turnaround time by orders of magnitude. We anticipate direct impact on simulation workflows through intelligent, scalable decomposition of complex CAD models into hex-meshable subdomains.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Closing the Loop between In Situ Stress Complexity and EGS Fracture Complexity

42 ENGINEERING↗

DOE OSTI2025

REBOUND: Reverse Engineering Bidirectional Outflow Under Non-Equilibrium Diffusion

Rare-earth elements (REEs) are essential for electronics, renewable energy, and defense technologies. However, the current supply of REEs relies on mining concentrated in a few countries and energy-intensive separations. DOE’s Basic Energy Sciences (BES) program has launched a grand challenge which aims to ensure a sustainable supply of critical REEs by developing innovative and environmentally friendly separation methods. As an alternative to costly and harmful traditional methods, the Non-Equilibrium Transport Driven Separations (NETS) initiative has created a microfluidic Y-channel co-flow method that applies external fields to exploit magneto- and electrohydrodynamic effects for separating dilute REE ions from complex feedstocks. Computational fluid dynamics (CFD) studies have identified a few operating conditions with promising ion selectivity and separation efficiency. However, challenges remain regarding Y-channel versatility across feedstocks and accurate incorporation of physical phenomena into CFD models. In this work, we develop a multi-fidelity modelling approach which integrates experimental results with CFD simulation to build a surrogate model for the dependence of separation efficiency to variation of design parameters. The surrogate model enables a reinforcement learning (RL) method to adaptively launch CFD and experimental runs, improving model fidelity around optimal Y-channel parameters.

36 MATERIALS SCIENCE↗

DOE OSTI2026

Numerical Modeling & Optimization of the iProTech Pitching Inertial Pump (PIP) Wave Energy Converter (WEC) (CRADA Final Report)

This project represents a continuation of the collaboration between iProTech and NLR to simulate, optimize and design the iProTech Pitching Inertial Pump (PIP) device. The objectives of this TEAMER project are twofold: 1. Refining the physical characteristics of the existing iProTech PIP WEC-Sim model to enhance the model’s fidelity and include controllable components. Key model enhancements target the inclusion of Coulomb friction, the introduction of a controllable bypass valve, and the replacement of traditional check valves with advanced motorized ones. 2. Exploring traditional and advanced control algorithms. From traditional methods like latching control to cutting-edge reinforcement learning (RL) algorithms, the goal is to ensure the PIP device's adaptability and optimal performance across a range of ocean conditions. NLR is tasked with augmenting the WEC-Sim model and implementing the control algorithms, culminating in performance comparison analyses. iProTech will update their existing 3D models, advise on model improvements, and determine crucial system metrics. WEC-Sim, developed in MATLAB/SIMULINK with Simscape Multibody, is the main piece of software that will be used in this project. Coupled with the MATLAB RL Toolbox, it offers a robust platform for in-depth simulation and optimization of the iProTech PIP device. Building on previous work to explore the PIP design space and optimize its geometry, mass distribution, center of gravity and other key parameters, this project aims to refine iProTech’s existing numerical models and develop effective control algorithms that can seamlessly integrate into their future hardware testing campaigns.

16 TIDAL AND WAVE POWER↗

DOE OSTI2025

Employing MACS/ViBRANT as a Surrogate MARVEL Reactor for Startup Reactivity Tuning and Supervisory Control Processes

Advanced nuclear reactors are a key part of the future of nuclear energy both in the United States and globally. They offer unique benefits for various energy-demanding applications, including use in remote locations, compact size, modular manufacturing, remote monitoring, low and/or variable power rating operation, and reliance on novel technologies to enhance operational safety. To achieve economic feasibility, advanced reactors must significantly reduce their workforces in comparison with the current fleet. Achieving this reduction will occur through reducing staff workloads using technology to achieve autonomous or semi-autonomous operations, demonstrated by comprehensive testing and validation activities. These operations will require both software and hardware platforms during the design and testing phases. While simulations are useful during the design phase, their performance can significantly deviate during actual deployment on hardware. This report presents the outcomes of a collaborative technical initiative between the U.S. Department of Energy (DOE) Microreactor Program (MRP) and Advanced Sensors and Instrumentation (ASI) Program. The collaboration utilized the Microreactor Automated Control System (MACS) hardware platform to bridge the gap between theoretical reactor design and actual startup and control operations. Two key use cases were investigated: facilitating the startup testing period and demonstrating supervisory control. The first use case details the key Microreactor Applications Research Validation and Evaluation (MARVEL) reactor startup physics testing activities conducted using the MACS platform. These activities included drum worth measurements, shutdown margin assessment, temperature feedback analysis, and scram time evaluation, as well as unique testing that would apply to the MARVEL reactor to demonstrate the testing methodologies in a low-risk environment. The MACS platform, serving as a surrogate representation of the MARVEL reactor, proved instrumental in performing these tests. The exercise revealed aspects that led to optimized processes, refined hardware design, and enhanced base software capabilities. By maturing methods and technologies in this manner, the initiative promises to reduce wasted time in the actual on-site reactor deployment effort, thereby saving significant time and resources. The second use case focuses on the development and implementation of supervisory control methods aimed at managing core tilt, which can result from asymmetrical operations or manufacturing imperfections in fuel rods or reactivity control devices. A key objective was to assess and compare the use of artificial intelligence (AI) for supervisory control. The effort aimed to define the role of supervisory control to enhance performance without risking control instability. This effort explored three distinct approaches: rules-based (RB) methods, optimization techniques, and reinforcement learning (RL) algorithms. Each approach was evaluated for its ease of implementation, its usability, and its effectiveness in responding to asymmetries in neutron flux. Comparative analysis of these approaches provided valuable insights into their applicability and effectiveness, offering a robust framework for advanced reactor operations. Together, these two use cases highlight the potential of hardware test beds to help streamline the design, operation, and control of advanced nuclear reactors. This collaborative effort underscores the importance of continued innovation and experimentation in achieving the next generation of safe, reliable, and economically viable nuclear energy solutions.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2026

Optimization and stabilization of Fermilab Booster using hybrid Bayesian/RL framework

PIPII project will raise Fermilab Booster intensity and ramp rate. Beam losses will limit average power and are hard to simulate. Presently, Booster uses operator-guided empirical tuning. This task is challenging due to high dimensionality, multiple objectives, critical safety constraints, and drifts. We developed a synergistic suite of Bayesian optimization (BO) and reinforcement learning (RL) tools to optimize and stabilize beam losses. First, active learning was used to build a rough model. Data was collected parasitically using two novel safety constraint types – nonlinear input space restrictions (based on optics model), and uncertainty constraints (to stop bad steps/beam aborts). We then applied online multi-objective BO with scalarized objectives and fitting to improve/rebalance losses, increasing safety margins by 25%. Using BO model as a safety veto, we tried several on/off-policy RL agents for long term stabilization; SAC had best performance. We found that adding contextual (state) information further improved performance, eventually integrating key knobs like linac phase and temperature into the parameter space. Long term testing is ongoing to enable operational use.

Kuklev, Nikita [Fermilab]↗

DOE OSTI2026

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗

DOE OSTI2025

GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

Large language models (LLMs) are increasingly adapted to downstream tasks via reinforcement learning (RL) methods like Group Relative Policy Optimization (GRPO), which often require thousands of rollouts to learn new tasks. We argue that the interpretable nature of language often provides a much richer learning medium for LLMs, compared to policy gradients derived from sparse, scalar rewards. To test this, we introduce GEPA (Genetic-Pareto), a prompt optimizer that thoroughly incorporates natural language reflection to learn high-level rules from trial and error. Given any AI system containing one or more LLM prompts, GEPA samples trajectories (e.g., reasoning, tool calls, and tool outputs) and reflects on them in natural language to diagnose problems, propose and test prompt updates, and combine complementary lessons from the Pareto frontier of its own attempts. As a result of GEPA's design, it can often turn even just a few rollouts into a large quality gain. Across six tasks, GEPA outperforms GRPO by 6% on average and by up to 20%, while using up to 35x fewer rollouts. GEPA also outperforms the leading prompt optimizer, MIPROv2, by over 10% (e.g., +12% accuracy on AIME-2025), and demonstrates promising results as an inference-time search strategy for code optimization. We release our code at https://github.com/gepa-ai/gepa.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Flexible Reinforcement Learning Framework for Building Control using EnergyPlus-Modelica Energy Models

In recent years, reinforcement learning (RL) methods have been greatly enhanced by leveraging deep learning approaches. RL methods applied to building control have shown potential in many applications due to their ability to complement or replace conventional methods such as model-based or rule-based controls. However, RL-based building control software is likely tailored either to one target building system or to a specific RL method so that significant additional effort would be required to customize the RL-based controller for use in other building systems or with other RL approaches. Also, RL-based building controls usually depend on building energy simulations to train controllers, so emulating building dynamics (i.e., thermal dynamics and control dynamics) and capturing sub-hourly dynamic profiles are crucial to further the development of effective RL-based building control methods. To address these challenges, we present an open source RL-based control software employing a high-fidelity hybrid EnergyPlus-Modelica building energy model which emulates building dynamics at 1-minute resolution. This software consists of decoupled components (environment, building emulator, control agent, and RL algorithm), which allows for quick prototyping and benchmarking of standard RL algorithms in different systems; for example, a single component can be replaced without revising all of the software. To demonstrate this software framework, we conducted a benchmark study using an EnergyPlus-Modelica building energy model for a Chicago office building with an RL-based controller to dynamically control the chilled water temperature setpoint and the air handling unit supply air temperature setpoint on selected floors.

Lee, Joon-Yong↗

DOE OSTI2020

Automated Adversary Emulation for Cyber-Physical Systems via Reinforcement Learning

Adversary emulation is an offensive exercise that provides a comprehensive assessment of a system’s resilience against cyber attacks. However, adversary emulation is typically a manual process, making it costly and hard to deploy in cyber-physical systems (CPS) with complex dynamics, vulnerabilities, and operational uncertainties. In this paper, we develop an automated, domain-aware approach to adversary emulation for CPS. We formulate a Markov Decision Process (MDP) model to determine an optimal attack sequence over a hybrid attack graph with cyber (discrete) and physical (continuous) components and related physical dynamics. We apply model-based and model-free reinforcement learning (RL) methods to solve the discrete-continuous MDP in a tractable fashion. As a baseline, we also develop a greedy attack algorithm and compare it with the RL procedures. We summarize our findings through a numerical study on sensor deception attacks in buildings to compare the performance and solution quality of the proposed algorithms.

Bhattacharya, Arnab↗

DOE OSTI2020

Single Episode Policy Transfer in Reinforcement Learning

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, we report our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

42 ENGINEERING↗

DOE OSTI2021

The Effect of Antagonistic Behavior in Reinforcement Learning

The significant achievements of deep reinforcement learning (RL) have motivated researchers to also investigate its shortcomings. Such work has shown that typical methods in deep RL tend to produce brittle policies that overfit to the training environment. In this paper, we introduce the notion of purely antagonistic behavior in value-based agents, where the objective is not to maximize reward but to minimize the victim’s value over time. This notion is motivated by the scenario in which an antagonistic human architect, without access to the environment’s reward function, wants to build an RL agent that can impede another well-trained RL victim agent. First, we formalize a notion of antagonistic behavior in RL. Then, we provide experiments that show how a purely antagonistic agent performs compared to a well-trained victim that learns directly from the game’s rewards. Our results suggest that if one’s goal is to find vulnerabilities in well-trained agents, direct access to the environment’s rewards is not necessary, and antagonistic behavior can be measured independently from environment wins and losses.

Fujimoto, Ted C.↗

DOE OSTI2022

Learning infinite-horizon average-reward restless multi-action bandits via index awareness

We consider the online restless bandits with average-reward and multiple actions, where the state of each arm evolves according to a Markov decision process (MDP), and the reward of pulling an arm depends on both the current state of the corresponding MDP and the action taken. Since finding the optimal control is typically intractable for restless bandits, existing learning algorithms are often computationally expensive or with a regret bound that is exponential in the number of arms and states. In this paper, we advocate \textit{index-aware reinforcement learning} (RL) solutions to design RL algorithms operating on a much smaller dimensional subspace by exploiting the inherent structure in restless bandits. Specifically, we first propose novel index policies to address dimensionality concerns, which are provably optimal. We then leverage the indices to develop two low-complexity index-aware RL algorithms, namely, (i) GM-R2MAB, which has access to a generative model; and (ii) UC-R2MAB, which learns the model using an upper confidence style online exploitation method. We prove that both algorithms achieve a sub-linear regret that is only polynomial in the number of arms and states. A key differentiator between our algorithms and existing ones stems from the fact that our RL algorithms contain a novel exploitation that leverages our proposed provably optimal index policies for decision-makings.

Xiong, Guojun↗

DOE OSTI2023

Self-Driving Telescopes: Autonomous Scheduling of Astronomical Observation Campaigns with Offline Reinforcement Learning

Modern astronomical experiments are designed to achieve multiple scientific goals, from studies of galaxy evolution to cosmic acceleration. These goals require data of many different classes of night-sky objects, each of which has a particular set of observational needs. These observational needs are typically in strong competition with one another. This poses a challenging multi-objective optimization problem that remains unsolved. The effectiveness of Reinforcement Learning (RL) as a valuable paradigm for training autonomous systems has been well-demonstrated, and it may provide the basis for self-driving telescopes capable of optimizing the scheduling for astronomy campaigns. Simulated datasets containing examples of interactions between a telescope and a discrete set of sky locations on the celestial sphere can be used to train an RL model to sequentially gather data from these several locations to maximize a cumulative reward as a measure of the quality of the data gathered. We use simulated data to test and compare multiple implementations of a Deep Q-Network (DQN) for the task of optimizing the schedule of observations from the Stone Edge Observatory (SEO). We combine multiple improvements on the DQN and adjustments to the dataset, showing that DQNs can achieve an average reward of 87%+-6% of the maximum achievable reward in each state on the test set. This is the first comparison of offline RL algorithms for a particular astronomical challenge and the first open-source framework for performing such a comparison and assessment task.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2023

Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e

We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab). Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale. We treat the Mu2e accelerator system as a Markov Decision Process suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and enhance training stability. A key innovation in our approach is the integration of a neuralized Proportional-Integral-Derivative (PID) controller into the policy function, resulting in a significant improvement in the Spill Duty Factor (SDF) by 13.6%, surpassing the performance of the current PID controller baseline by an additional 1.6%. This paper presents the preliminary offline results based on a differentiable simulator of the Mu2e accelerator. It paves the groundwork for real-time implementations and applications, representing a crucial step towards automated proton beam intensity control for the Mu2e experiment.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory

Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.

Wang, Yu↗

DOE OSTI2025

LC-Opt: Benchmarking Reinforcement Learning and Agentic AI for End-to-End Liquid Cooling Optimization in Data Centers

Liquid cooling is critical for thermal management in high-density data centers with the rising AI workloads. However, machine learning-based controllers are essential to unlock greater energy efficiency and reliability, promoting sustainability. We present LC-Opt, a Sustainable Liquid Cooling (LC) benchmark environment, for reinforcement learning (RL) control strategies in energy-efficient liquid cooling of high-performance computing (HPC) systems. Built on the baseline of a high-fidelity digital twin of Oak Ridge National Lab's Frontier Supercomputer cooling system, LC-Opt provides detailed Modelica-based end-to-end models spanning site-level cooling towers to data center cabinets and server blade groups. RL agents optimize critical thermal controls like liquid supply temperature, flow rate, and granular valve actuation at the IT cabinet level, as well as cooling tower (CT) setpoints through a Gymnasium interface, with dynamic changes in workloads. This environment creates a multi-objective real-time optimization challenge balancing local thermal regulation and global energy efficiency, and also supports additional components like a heat recovery unit (HRU). We benchmark centralized and decentralized multi-agent RL approaches, demonstrate policy distillation into decision and regression trees for interpretable control, and explore LLM-based methods that explain control actions in natural language through an agentic mesh architecture designed to foster user trust and simplify system management. LC-Opt democratizes access to detailed, customizable liquid cooling models, enabling the ML community, operators, and vendors to develop sustainable data center liquid cooling control solutions.

Naug, Avisek [Hewlett Packard Enterprise]↗

DOE OSTI2026

Intern Poster Session 08/13: Autonomous Nuclear Robotics: Applications in nuclear waste inspection and hot cell experiments

The nuclear industry is experiencing renewed interest in autonomous robotics, yet most deployed systems remain teleoperated with limited autonomy. This work presents two contributions toward fully autonomous nuclear robotic systems: autonomous waste inspection at the Hanford Site and an autonomous hot cell laboratory framework. Inspections of Hanford's underground waste storage tanks are performed manually at significant cost and personnel exposure. We developed a reinforcement-learning (RL) training pipeline for a custom-built inspection arm. In parallel, we are designing an autonomous laboratory framework for post-irradiation examination in hot cells at the Specimen Preparation Laboratory (SPL) that integrates computer vision, task and motion planning, hardware execution, and operator-in-the-loop control. These systems demonstrate a path toward safer, more efficient nuclear operations by reducing human exposure while maintaining rigorous human oversight at critical decision points.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗