Search NASA⌕ Search

SEARCH · Search NASA

Results for “Q learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Decentralised Reinforcement Learning for Dynamic Cyberattack Response in Microgrid Networks

Microgrids rely on communication networks for reliable operation, which makes them inherently vulnerable to cyberattacks. Such attacks can destabilise system dynamics and drive states away from their nominal operating trajectories. Although several physics-informed and machine learning-based strategies have been developed to counter these threats, the rapidly evolving cyber landscape enables adversaries to bypass static defences or rules-based mitigation approaches. This paper proposes a dynamic, online-trained and fully decentralised reinforcement learning (RL)-based cyberattack response framework to protect microgrids from evolving cyberattacks. The proposed framework deploys multiple deep Q-networks (DQNs), each associated with a distributed energy resource (DER), to enable localised and adaptive attack mitigation. In this framework, each DQN processes local voltage and frequency measurements—combined with intrusion detection system (IDS) alerts—as observations and rewards to guide decision-making. Extensive simulation studies demonstrate the robustness of the proposed framework under diverse attack scenarios and varying IDS-induced detection delays. Comparative analysis highlights its superiority over existing static or preexisting rules-based mitigation approaches. Finally, we present an analysis that shows the framework's scalability to real-life microgrids with more interacting agents.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1999

Lesson Plan Prototype for International Space Station's Interactive Video Education Events

The outreach and education components of the International Space Station Program are creating a number of materials, programs, and activities that educate and inform various groups as to the implementation and purposes of the International Space Station. One of the strategies for disseminating this information to K-12 students involves an electronic class room using state of the art video conferencing technology. K-12 classrooms are able to visit the JSC, via an electronic field trip. Students interact with outreach personnel as they are taken on a tour of ISS mockups. Currently these events can be generally characterized as: Being limited to a one shot events, providing only one opportunity for students to view the ISS mockups; Using a "one to many" mode of communications; Using a transmissive, lecture based method of presenting information; Having student interactions limited to Q&A during the live event; Making limited use of media; and Lacking any formal, performance based, demonstration of learning on the part of students. My project involved developing interactive lessons for K-12 students (specifically 7th grade) that will reflect a 2nd generation design for electronic field trips. The goal of this design will be to create electronic field trips that will: Conform to national education standards; More fully utilize existing information resources; Integrate media into field trip presentations; Make support media accessible to both presenters and students; Challenge students to actively participate in field trip related activities; and Provide students with opportunities to demonstrate learning

Zigon, Thomas↗
NASA NTRSDate not supplied

Design and Analysis of Convolutional Neural Network for RF Signal Modulation Classification for In-Orbit Deployment

To effectively transmit data to and from satellites requires a complex and robust RF communication system. Commonly, several different types of signal modulations may be required to maximize satellite efficiency depending on a variety of unexpected channel impairments. We propose a neural network algorithm capable of learning these RF signal modulations using a supervised learning technique designed for low power, high-efficiency in-orbit deployment. The work presented demonstrates a convolutional neural network (CNN) capable of learning and recognizing a set of modulation schemes commonly used to transmit RF information. We are capable of recognizing the modulation scheme from the I and Q data channels directly, with no preprocessing or data conversion required other than breaking the incoming signal into a set of uniform normalized samples. We perform a network design and size analysis, showing that reasonably high accuracy can be obtained using networks with a relatively low number of trainable parameters. Given that a user of a system such as this may wish to receive a signal using a modulation scheme that the network has not previously learned, we demonstrate that transfer learning can learn new modulation schemes by retraining only the fully connected layers in the CNN. Thus, this type of network would excel in outer space deployment using high-efficiency transfer learning hardware. Modulation recognition can be performed through rapid feedforward computation, and the CNN training process is significantly simplified when learning new modulations is required.

CNN↗
DOE OSTI2025

Meeting Global Health Needs via Infectious Disease Forecasting: Development of a Reliable Data-Driven Framework

Infectious diseases (IDs) have a significant detrimental impact on global health. Timely and accurate ID forecasting can result in more informed implementation of control measures and prevention policies. To meet the operational decision-making needs of real-world circumstances, we aimed to build a standardized, reliable, and trustworthy ID forecasting pipeline and visualization dashboard that is generalizable across a wide range of modeling techniques, IDs, and global locations. We forecasted 6 diverse, zoonotic diseases (brucellosis, campylobacteriosis, Middle East respiratory syndrome, Q fever, tick-borne encephalitis, and tularemia) across 4 continents and 8 countries. We included a wide range of statistical, machine learning, and deep learning models (n=9) and trained them on a multitude of features (average n=2326) within the One Health landscape, including demography, landscape, climate, and socioeconomic factors. The pipeline and dashboard were created in consideration of crucial operational metrics—prediction accuracy, computational efficiency, spatiotemporal generalizability, uncertainty quantification, and interpretability—which are essential to strategic data-driven decisions. While no single best model was suitable for all disease, region, and country combinations, our ensemble technique selects the best-performing model for each given scenario to achieve the closest prediction. For new or emerging diseases in a region, the ensemble model can predict how the disease may behave in the new region using a pretrained model from a similar region with a history of that disease. The data visualization dashboard provides a clean interface of important analytical metrics, such as ID temporal patterns, forecasts, prediction uncertainties, and model feature importance across all geographic locations and disease combinations. As the need for real-time, operational ID forecasting capabilities increases, this standardized and automated platform for data collection, analysis, and reporting is a major step forward in enabling evidence-based public health decisions and policies for the prevention and mitigation of future ID outbreaks.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

Decentralized Distributed Proximal Policy Optimization (DD-PPO) for High Performance Computing Scheduling on Multi-User Systems

Resource allocation in High Performance Computing (HPC) environments presents a complex and multifaceted challenge for job scheduling algorithms. Beyond the efficient allocation of system resources, schedulers must account for and optimize multiple performance metrics, including job wait time and system throughput. Traditional heuristic-based scheduling algorithms increasingly struggle and lack the efficiency needed to meet the demands and address the complexity and scale of modern HPC systems. Consequently, recent research efforts have focused on leveraging advancements in Artificial Intelligence (AI) and Deep Learning (DL), particularly Reinforcement Learning (RL), to develop more adaptable and intelligent scheduling strategies. Previous RL-based scheduling approaches have explored a range of algorithms, from Deep Q-Networks (DQN) to Proximal Policy Optimization (PPO), and more recently, hybrid methods that integrate Graph Neural Networks (GNNs) with RL techniques. However, a common limitation across these methods is their reliance on relatively small datasets, with few methods being evaluated using large-scale, multi-million-job trace datasets representative of real-world HPC workloads. Moreover, existing RL schedulers face scalability issues due to centralized policy updates, which hinder training efficiency and performance when applied to large datasets. This study introduces a novel RL-based scheduler utilizing Decentralized Distributed Proximal Policy Optimization (DD-PPO) algorithm, which supports large-scale distributed training across multiple workers without requiring parameter synchronization at every step. By eliminating reliance on centralized updates to a shared policy, the DD-PPO scheduler enhances scalability, training efficiency, and sample utilization. Experimental validation using a large real-world dataset containing over 11.5 million job traces collected from petascale HPC systems over six years assesses the influence of dataset scale on training effectiveness and compares DD-PPO performance to traditional and advanced scheduling approaches. The experimental results demonstrate improved scheduling performance in comparison to both heuristic-based schedulers and existing RL-based scheduling algorithms.

AI↗
NASA NTRS2017

A Machine Learning Concept for DTN Routing

This paper discusses the concept and architecture of a machine learning based router for delay tolerant space networks. The techniques of reinforcement learning and Bayesian learning are used to supplement the routing decisions of the popular Contact Graph Routing algorithm. An introduction to the concepts of Contact Graph Routing, Q-routing and Naive Bayes classification are given. The development of an architecture for a cross-layer feedback framework for DTN (Delay-Tolerant Networking) protocols is discussed. Finally, initial simulation setup and results are given.

Delay Tolerant Networks↗
DOE OSTI2025

A Dark Photon Dark Matter Search with a Widely-Tunable SRF Cavity

The SERAPH (SupERconducting Axion and Paraphoton Haloscope) experiment is a family of superconducting haloscopes being developed by the Superconducting Quantum Materials and Systems (SQMS) Center to search for wavelike dark matter. This presentation will focus on preliminary results from our dark photon dark matter search using a widely-tunable SRF cavity operating between 4-7 GHz, nicknamed the “plunger cavity.” I will present the cavity design and characterization, analyze the impact of microphonics on system performance and haloscope sensitivity, and describe our tuning methodology for this Q~10^8 cavity. The presentation will cover our haloscope analysis approach and discuss sensitivity limits achieved within this frequency range. Finally, I will outline lessons learned from this first search, implications for future axion searches, and proposed improvements for subsequent SERAPH experiments.

Cervantes, Raphael [Fermilab]↗
DOE OSTI2025

A Dark Photon Dark Matter Search with a Widely-Tunable SRF Cavity

The SERAPH (SupERconducting Axion and Paraphoton Haloscope) experiment is a family of superconducting haloscopes being developed by the Superconducting Quantum Materials and Systems (SQMS) Center to search for wavelike dark matter. This presentation will focus on preliminary results from our dark photon dark matter search using a widely-tunable SRF cavity operating between 4-7 GHz, nicknamed the "plunger cavity." I will present the cavity design and characterization, analyze the impact of microphonics on system performance and haloscope sensitivity, and describe our tuning methodology for this Q~10^8 cavity. The presentation will cover our haloscope analysis approach and discuss sensitivity limits achieved in this frequency range. Finally, I will outline lessons learned from this first search, implications for future axion searches, and proposed improvements for subsequent SERAPH experiments.

Cervantes, Raphael [Fermilab]↗
DOE OSTI2024

Optimal Coordination of Electric Vehicles for Grid Services using Deep Reinforcement Learning

Recent research has shown the effectiveness of reinforcement learning (RL) in coordinating electric vehicles (EVs) with vehicle-to-grid capabilities for grid services. However, many of these studies rely on lookup table and deep Q-network techniques, which can be impractical when dealing with continuous states and actions. In addition, existing RL designs inadequately account for battery aging effects, EV user satisfaction, uncertain departure and arrival time, and trip distance, which may compromise effective coordination. This paper aims to bridge these gaps by developing an innovative deep deterministic policy gradient-based RL framework for optimal coordination of EVs. Case studies were carried out using a test system with 100 EVs, and numerical analysis results showed that the proposed RL framework can effectively coordinate EVs to maximize economic benefits and user satisfaction while ensuring the expected battery lifespan.

Das, Avijit↗
DOE OSTI2024

Quantum Circuit Partitioning for Scalable Noise-Aware Quantum Circuit Re-Synthesis

Re-synthesis techniques are utilized to optimize the quantum circuit. To enable scalable re-synthesis a divide-and-conquer approach is adopted that partitions the circuit into smaller blocks, which are optimized independently. Several algorithms have been proposed to minimize the block number while maximizing the gate count of each block. However, they vary in their performance and may not yield the highest output fidelity. We propose a reinforcement learning-based quantum circuit partitioning framework that incorporates the physical properties of the quantum hardware to maximize the output fidelity post-quantum circuit optimization. To accelerate the training, we also propose a noise injection method that enables on-the-fly optimization in the reinforcement learning environment, independent of the adopted optimization/re-synthesis method at the block level. We evaluate our approach compared to different partitioning techniques using various quantum benchmarks executed on IBM Q Hanoi quantum computer.

Charrwi, Mohammad Walid↗
NASA NTRS2024

Piezoelectric impedance-based high-accuracy damage identification using sparsity conscious multi-objective optimization inverse analysis

Two elements are essential in structural health monitoring utilizing dynamic responses: response measurement with high-frequency contents, i.e., small characteristic wavelengths, that can adequately reflect damage features, and effective inverse identification analysis that is however oftentimes under-determined. The advancement of smart structure integration has led to active interrogation through frequency-sweeping piezoelectric impedance measurement at high frequency range. In this research we develop a multi-objective optimization formulation for the identification of damage location and severity utilizing piezoelectric impedance. While one optimization objective is to match the response measurement with finite element model prediction in the damage parametric space, the other is the number of locations of damage, i.e., the sparsity of damage index as the solution vector, since damage usually occurs within a small number of locations. This multi-objective formulation fits well the under-determined nature of damage identification, as it naturally provides multiple solutions as basis for further elucidation. The challenge remaining is how to find a small solution set that can include the actual damage scenario. Here we develop a novel inverse identification framework utilizing the intelligent swarm optimizer which possesses flexibility for enhancement. We first embed a sparsity enforcement process into the population generation of the optimizer, which yields a solution repository intrinsically possessing sparsity. We then apply reinforcement learning so the agents can adaptively opt for local strategies with the aim of enriching the searching patterns to diversify the solutions. Through the incorporation of a Q-table, searching toward more promising directions will be rewarded. Our case analyses employing experimental data indicate that this sparsity-conscious multi-objective particle swarm optimization technique can lead to a small solution set which generally encompasses the true damage scenario. This effectively solves the structural damage identification problem with piezoelectric impedance measurement.

Yang Zhang↗
DOE OSTI2025

Confinement performance predictions for a high field axisymmetric tandem mirror

This paper presents a Hammir tandem mirror confinement performance analysis based on Realta Fusion’s first-of-a-kind model for axisymmetric magnetic mirror fusion performance. This model uses an integrated end plug simulation model including, heating, equilibrium and transport combined with a new formulation of the plasma operation contours (POPCONs) technique for the tandem mirror central cell. Using this model in concert with machine learning optimization techniques, it is shown that an end plug utilizing high temperature superconducting magnets and modern neutral beams enables a classical tandem mirror pilot plant producing a fusion gain Q > 5. The approach here represents an important advance in tandem mirror design. The high-fidelity end plug model enables calculations of heating and transport in the highly non-Maxwellian end plug to be made more accurately. The detailed end plug modelling performed in this work has highlighted the importance of classical radial transport and neutral beam absorption efficiency on end plug viability. The central cell POPCON technique allows consideration of a wide range of parameters in the relatively simple near-Maxwellian central cell, facilitating the selection of more optimal central cell plasmas. These advances make it possible to find more conservative classical tandem mirror fusion pilot plant operating points with lower temperatures, neutral beam energies and end plug performance requirements than designs in the literature. Despite being more conservative, it is shown that these operating points have sufficient confinement performance to serve as the basis of a viable fusion pilot plant provided that they can be stabilized against magnetohydrodynamic and trapped particle modes.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Hierarchical Reinforcement Learning of a Short-Range Bond-Order Potential for Silica: Analytic Embedding of Coordination with Classical Efficiency

Reinforcement learning (RL) has recently emerged as a data-efficient strategy to parametrize short-range interatomic potentials. Building on our past RL optimization of pairwise silica models, we extend the framework to a bond-order (Tersoff-type) potential that provides an analytic embedding of local coordination through a three-body term. A hierarchical RL workflow combining continuous-action Monte Carlo Tree Search and property-based rewards efficiently explores the 26-dimensional parameter space, sequentially optimizing lattice parameters, densities, angles, and cohesive energies of 21 silica polymorphs. The resulting models, Q-Tersoff and ML-Tersoff, reproduce the energetic ordering of low-energy phases and capture the angular correlations and amorphous structure factors of silica with improved fidelity over pairwise force fields, while remaining orders of magnitude faster than high-dimensional machine-learned potentials. Both models underperform for elastic constants and high-energy frameworks, delineating the limits of the current analytic form. The approach establishes a general and interpretable route to angle-aware, short-range potentials that bridge physics-based and machine-learned descriptions of silicate materials.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Forming a database to study reversed magnetic shear from the National Spherical Torus eXperiment using machine learning

Achieving a long-lived reversed magnetic shear (RMS) target plasma in the National Spherical Torus eXperiment Upgrade will require developing various sustainment scenarios. To help with the ongoing plasma control efforts, the development of a new analysis for the motional Stark effect (MSE) diagnostic using a machine learning algorithm, namely, MSE-ML, is described. MSE-ML will be used to identify patterns during RMS discharges, some of which suffer magnetohydrodynamic (MHD) events resulting in current redistribution and monotonic q-profiles. A database consisting of q and magnetic shear profiles is being constructed primarily based on the existing National Spherical Torus eXperiment data with equilibrium reconstructions constrained by the magnetic field pitch angle profile measured using the multi-channel MSE diagnostic. An unsupervised k-means clustering of the data is developed to study the RMS formation as a function of time. The initial clustering from the q-profiles shows significant differences in both amplitude and the duration of the RMS period. As a goal, the clustering results that detect and distinguish shots with substantial and sustained RMS are to be used as a preprocessing step in a supervised algorithm to identify the underlying conditions that lead to long-lasting improved confinement with RMS. Another aim of the MSE-ML study is to identify precursors of RMS-destroying MHD events in either derived data such as the q-profile or directly measured data such as the magnetic field pitch angle profile.

Uzun-Kaymak, I. U. (ORCID:0000000276251493)↗
DOE OSTI2024

Predicting Partial Atomic Charges in Metal–Organic Frameworks: An Extension to Ionic MOFs

Molecular simulation is an invaluable tool to predict and understand the usage of metal–organic frameworks (MOFs) for gas storage and separation applications. Accurate partial atomic charges, commonly obtained from density functional theory (DFT) calculations, are often required to model the electrostatic interactions between the MOF and adsorbates, especially when the adsorbates have dipole or quadrupole moments, such as water and CO 2 . Machine learning (ML) models have been previously employed to predict partial charges and avoid the computational cost associated with DFT calculations. However, previous ML models suffer from small training data sets, which limit their scope of application. In this work, we introduce two novel machine learning models, PACMOF2-neutral and PACMOF2-ionic, aimed at predicting the density-derived electrostatic and chemical (DDEC6) partial atomic charges for both neutral and ionic MOFs. These models not only yield DFT-level accuracy at a fraction of the computational cost but also demonstrate a remarkable improvement in prediction of adsorption, as validated with grand canonical Monte Carlo simulations. Furthermore, the robustness and fast computational time of the PACMOF2 models, along with their transferability to other porous materials such as covalent organic frameworks and zeolites, underscores their potential in high-throughput screening of MOFs for diverse applications.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Using convolutional neural networks to detect edge localized modes in DIII-D from Doppler backscattering measurements

In H-mode tokamak plasmas, the plasma is sometimes ejected beyond the edge transport barrier. These events are known as edge localized modes (ELMs). ELMs cause a loss of energy and damage the vessel walls. Understanding the physics of ELMs, and by extension, how to detect and mitigate them, is an important challenge. In this paper, we focus on two diagnostic methods—deuterium-alpha (D α ) spectroscopy and Doppler backscattering (DBS). The former detects ELMs by measuring Balmer alpha emission, while the latter uses microwave radiation to probe the plasma. DBS has the advantages of having a higher temporal resolution and robustness to damage. These advantages of DBS diagnostic may be beneficial for future operational tokamaks, and thus, data processing techniques for DBS should be developed in preparation. In sight of this, we explore the training of neural networks to detect ELMs from DBS data, using D α data as the ground truth. With shots found in the DIII-D database, the model is trained to classify each time step based on the occurrence of an ELM event. The results are promising. When tested on shots similar to those used for training, the model is capable of consistently achieving a high f1-score of 0.93. Furthermore, this score is a performance metric for imbalanced datasets that ranges between 0 and 1. We evaluate the performance of our neural network on a variety of ELMs in different high confinement regimes (grassy ELM, RMP mitigated, and wide-pedestal), finding broad applicability. Beyond ELMs, our work demonstrates the wider feasibility of applying neural networks to data from DBS diagnostic.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

CoRE MOF DB: A curated experimental metal-organic framework database with machine-learned properties for integrated material-process screening

Here, we present an updated version of the Computation-Ready, Experimental (CoRE) Metal-Organic Framework (MOF) database, which includes a curated set of computation-ready MOF crystal structures designed for high-throughput computational materials discovery. Data collection and curation procedures were improved from the previous version to enable more frequent updates in the future. Machine-learning-predicted properties, such as stability metrics and heat capacities, are included in the dataset to streamline screening activities. An updated version of MOFid was developed to provide detailed information on metal nodes, organic linkers, and topologies of an MOF structure. DDEC6 partial atomic charges of MOFs were assigned based on a machine-learning model. Gibbs ensemble Monte Carlo simulations were used to classify the hydrophobicity of MOFs. The finalized dataset was subsequently used to perform integrated material-process screening for various carbon-capture conditions using high-fidelity temperature-swing adsorption (TSA) simulations. Our workflow identified multiple MOF candidates that are predicted to outperform CALF-20 for these applications.

CoRE MOF database↗