SEARCH · Search NASA

Results for “Reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

All sources NASA NTRS DOE OSTI

DOE OSTI2022

An Intelligent Distributed Ledger Construction Algorithm for IoT

Blockchain is the next generation of secure data management that creates near-immutable decentralized storage. Secure cryptography created a niche for blockchain to provide alternatives to well-known security compromises. However, design bottlenecks with traditional blockchain data structures scale poorly with increased network usage and are extremely computation-intensive. This made the technology difficult to combine with limited devices, like those in Internet of Things networks. In protocols like IOTA, replacement of blockchain's linked-list queue processing with a lightweight dynamic ledger showed remarkable throughput performance increase. However, current stochastic algorithms for ledger construction suffer distinct trade-offs between efficiency and security. This work proposed a machine-learning approach with a multi-arm bandit that resolved these issues and was designed for auditing on limited devices. This algorithm was tested in a reinforcement-learning environment simulating the IOTA ledger's construction with a decision tree. This study showed through regret analysis and experimentation that this approach was secure against impulse manipulation attacks while remaining energy-efficient. Although the IOTA protocol was a pioneer for lightweight distributed ledgers, it is expected that future blockchain protocols will adopt techniques similar to those presented in this work.

multi-arm bandit↗

DOE OSTI2025

A Dynamic Pricing Method to Manage the Impact of EV Charging on the Grid Using RL

This work addresses the challenge of managing electrical vehicle (EV) charging loads on distribution feeders with the increase in deployment of fast charging stations. To mitigate the adverse impacts on feeder health, a novel dynamic grid-informed pricing approach is proposed. This approach leverages reinforcement learning (RL) to determine hourly charging prices based on real-time grid conditions. A synthetic environment was developed to train the reinforcement learning agent. A model of an IEEE 34-bus distribution feeder with EV charging stations has been developed in OpenDSS utilizing Caldera for realistic EV charging profiles. Test cases demonstrate that the dynamic pricing strategy achieves higher energy delivery to the EV end user at a lower cost compared to constant pricing methods, while lowering voltage deviations and congestion. This approach offers more granular price adjustments, responding dynamically to feeder conditions and potentially improving grid stability and efficiency. The communication architecture to implement this dynamic pricing method is described. This research contributes to the development of smart grid-informed charging solutions that can reduce the cost of charging to the end user while also helping the grid.

EV charging, dynamic pricing, grid-informed chargi↗

DOE OSTI2025

A dynamic pricing method to manage the impact of EV charging on the grid using RL

24 - POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2025

blastforge

BlastForge is a python module meant to aid reinforcement learning research for geometry optimization research projects. The code is built to use PyTorch as a backend and will contain several model architectures and reinforcement learning training loops as well as helper python functions to evaluate a model’s performance during and after training. BlastForge is meant to be a small, focused python project to study moderate-complexity geometry optimization problems

Hickmann, Kyle [Los Alamos National Laboratory]↗

DOE OSTI2019

Learning Interactions in Complex Biological Systems

In this project, we developed novel methods for integrating simulation and data-driven methods to accelerate biomedical discovery. In particular, we exploit agent-based models of disease processes and deep reinforcement learning to identify potential novel multi-drug, patient-specific, and adaptive therapeutic strategies. We use the Innate Immune Response agent-based model (IIRABM) to model sepsis as a demonstration of our approach. We first calibrate the agent-based model using novel stochastic optimization algorithms develop in this project, then use the calibrated model to identify optimal cytokine mediation strategies using deep reinforcement learning (DRL). The learned policy achieves a dramatic reduction in mortality over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are three-fold: 1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, 2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and 3) optimized treatments respond to a patient’s individual disease progression over time, therefore capturing both differences across patients and the inherent randomness of disease progression within a single patient. We hope that this work motivates both considering adaptive, personalized multi-cytokine mediation therapy for sepsis and exploiting simulation with DRL for precision medicine more broadly.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2024

Advanced Computational Techniques for Improving Resilience of Critical Energy Infrastructure under Cyber-Physical Attacks

In this chapter, we present recent advances in improving the resilience of cyber-physical systems, especially with regards to energy systems. We provide discussions around various types of cyber-physical events that can cause disruptions and new advances in optimization, control, and reinforcement learning (RL) to deal with the challenges posed by such cyber-physical events. The presented methods range from distributed robust optimization, autonomous and coordinated control, reinforcement learning based resilient control and topology reconfiguration in Inter-System resilient control.

Nazir, Mohammad Nawaf [BATTELLE (PACIFIC NW LAB)]↗

DOE OSTI2019

Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities

Buildings account for 35% of the global final energy demand. Efficiency improvements and advanced control strategies have a significant impact in the reduction of energy costs and CO2 emissions. Building energy simulation is widely used to help planners, contractors, and building owners analyse diverse options regarding the planning and management of energy consumption in buildings. Furthermore, recent advances in data processing and computing have led to the development of sophisticated machine learning algorithms that can learn from large datasets, e.g., sensor data from buildings, and use them to develop building-specific adaptive and automatic energy controllers. Control algorithms, such as deep reinforcement learning can tune themselves, are model-free, and economical to implement. In this paper, we introduce an integrated simulation environment that combines CitySim, a fast building energy simulator, and TensorFlow, a platform for efficient implementation of advanced machine learning algorithms. The integration is achieved via Keras - an API for TensorFlow - and a set of text and csv files for data transfer between the applications. This new environment will allow researchers to investigate novel learning control algorithms, and demonstrate their robustness and potential for diverse applications in the built environment. We present two case studies for energy savings and demand response, respectively.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRSDate not supplied

ICE-RASSOR: Intelligent Capabilities Enhanced Regolith Advanced Surface Systems Operations Robot

NASA’s Regolith Advanced Surface Systems Operations Robot (RASSOR) is principally designed to mine and deliver regolith for In-Situ Resource Utilization (ISRU)processing. RASSOR’s design enables it to efficiently collect and deposit regolith, return collected material for processing, and myriad related ISRU activities. To reliably perform these operations on the lunar surface, RASSOR software and sensory systems need to be robust and maximize the information extracted from a reduced sensor payload. Herein, we present preliminary findings from the Intelligent Capabilities Enhanced RASSOR project. We apply supervised learning using real data to estimate the soil mass collected without the need for mass flow rate monitors or other explicate sensing techniques. We also create a reduced-order simulation environment to develop autonomous trenching controllers via reinforcement learning and prototype state estimation architectures. Our initial results suggest that excavated regolith mass can be inferred within 2.9% RMS error of full scale, and reinforcement learning for autonomous operations has learned viable trenching strategies and helped identify desirable sensing capabilities, arrangements, and considerations. Future work includes regolith mass estimation during dynamic operation, expanding our simulation to more complex environments, and transfer learning from simulation to hardware.

machine learning↗

NASA NTRSDate not supplied

ICE-RASSOR: Intelligent Capabilities Enhanced

NASA’s Regolith Advanced Surface Systems Operations Robot (RASSOR) is principally designed to mine and deliver regolith for In-Situ Resource Utilization (ISRU) processing. RAS-SOR’s design enables it to efficiently collect and deposit regolith, return collected material for processing, and myriad related ISRU activities. To reliably perform these operations on the lunar sur-face, RASSOR software and sensory systems need to be robust and maximize the information extracted from on-board sensing. Herein, we present preliminary findings from the Intelligent Capabilities Enhanced RASSOR project. We apply supervised learning using real data to estimate the soil mass collected without the need for mass flow rate monitors or other explicate sensing techniques. We also create a reduced-order simulation environment to develop autonomous trenching controllers via reinforcement learning and proto-type state estimation architectures. Our initial results suggest that excavated regolith mass can be inferred within 2.9% RMS error of full scale, and reinforcement learning for autonomous operations has learned viable trenching strategies and helped identify desirable sensing capabilities, arrangements, and considerations. Future work includes regolith mass estimation during dynamic operation, expanding our simulation to more complex environments, and transfer learning from simulation to hardware.

machine learning↗

NASA NTRSDate not supplied

ICE-RASSOR: Intelligent Capabilities Enhanced Regolith Advanced Surface Systems Operations Robot

NASA’s Regolith Advanced Surface Systems Operations Robot (RASSOR) is principally designed to mine and deliver regolith for In-Situ Resource Utilization (ISRU) processing. RASSOR’s design enables it to efficiently collect and deposit regolith, return collected material for processing, and myriad related ISRU activities. To reliably perform these operations on the lunar surface, RASSOR software and sensory systems need to be robust and maximize the information extracted from on-board sensory. Herein, we present preliminary findings from the Intelligent Capabilities Enhanced RASSOR project. We apply supervised learning using real data to estimate the soil mass collected without the need for mass flow rate monitors or other explicate sensing techniques. We also create a reduced-order simulation environment to develop autonomous trenching controllers via reinforcement learning and proto-type state estimation architectures. Our initial results suggest that excavated regolith mass can be inferred within 2.9% RMS error of full scale, and reinforcement learning for autonomous operations has learned viable trenching strategies and helped identify desirable sensing capabilities, arrangements, and considerations. Future work includes regolith mass estimation during dynamic operation, expanding our simulation to more complex environments, and transfer learning from simulation to hardware.

machine learning↗

DOE OSTI2023

Code for Value Decomposition Graph Network and environment for AMR on linear advection

This is the code for the paper [Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement](https://arxiv.org/abs/2211.00801), published at AAMAS 2023. It contains the implementation of a new algorithm, called Value Decomposition Graph Network (VDGN), for applying multi-agent reinforcement learning to the problem of adaptive mesh refinement (AMR). It also contains the implementation of a multi-agent environment for AMR on a linear advection problem. VDGN is the first learning algorithm to display anticipatory refinement behavior in AMR, and it outperforms local error threshold-based heuristic strategies.

Yang, Jiachen↗

DOE OSTI2018

Reinforced dynamics for enhanced sampling in large atomic and molecular systems

A new approach for efficiently exploring the configuration space and computing the free energy of large atomic and molecular systems is proposed, motivated by an analogy with reinforcement learning. There are two major components in this new approach. Like metadynamics, it allows for an efficient exploration of the configuration space by adding an adaptively computed biasing potential to the original dynamics. Like deep reinforcement learning, this biasing potential is trained on the fly using deep neural networks, with data collected judiciously from the exploration and an uncertainty indicator from the neural network model playing the role of the reward function. Parameterization using neural networks makes it feasible to handle cases with a large set of collective variables. This has the potential advantage that selecting precisely the right set of collective variables has now become less critical for capturing the structural transformations of the system. Finally, the method is illustrated by studying the full-atom explicit solvent models of alanine dipeptide and tripeptide, as well as the system of a polyalanine-10 molecule with 20 collective variables.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2023

OCHRE™ Gymnasium (ochre_gym) [SWR-23-47]

OCHRE™ Gymnasium is a Python framework for conducting reinforcement learning research on residential building energy management problems. It is a lightweight wrapper around OCHRE, a Python-based thermal-electric residential building simulator developed in-house at NREL. OCHRE Gymnasium adheres to the standard OpenAI Gym API for a reinforcement learning environment. Detailed Sphinx documentation will be auto-generated and accompany this codebase. This documentation will detail the API, provide examples of how to use the code, etc.

Emami, Patrick↗

DOE OSTI2018

Integrated Cyber/Physical Grid Resiliency Modeling

This project explored coupling modeling and analysis methods from multiple domains to address complex hybrid (cyber and physical) attacks on mission critical infrastructure. Robust methods to integrate these complex systems are necessary to enable large trade-space exploration including dynamic and evolving cyber threats and mitigations. Reinforcement learning employing deep neural networks, as in the AlphaGo Zero solution, was used to identify "best" (or approximately optimal) resilience strategies for operation of a cyber/physical grid model. A prototype platform was developed and the machine learning (ML) algorithm was made to play itself in a game of 'Hurt the Grid'. This proof of concept shows that machine learning optimization can help us understand and control complex, multi-dimensional grid space. A simple, yet high-fidelity model proves that the data have spatial correlation which is necessary for any optimization or control. Our prototype analysis showed that the reinforcement learning successfully improved adversary and defender knowledge to manipulate the grid. When expanded to more representative models, this exact type of machine learning will inform grid operations and defense - supporting mitigation development to defend the grid from complex cyber attacks! This same research can be expanded to similar complex domains.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

GOOML - Real World Applications of Machine Learning in Geothermal Operations

GOOML (Geothermal Operational Optimization with Machine Learning) is a machine-learning based framework that enables geothermal power plant operators to explore optimization opportunities for their assets in an efficient and robust digital environment. Backed by real-world data sources, thermodynamic constraints and steamfield intelligence, the GOOML environment provides new tools to explore how to best operate steamfields as well as test new scenarios and configurations prior to implementation in the field. To prove the effectiveness of GOOML, we have undertaken optimization experiments using reinforcement learning (RL) to generate operational suggestions using a balance of mass-take targets, sustainability considerations and net generation. Our experiments use the GOOML construct to explore different field parameters and perform multiple reinforcement learning experiments. Like a comprehensive laboratory workbench, we can change out components of a steamfield to perform testing under a variety of conditions (restrict mass, increase pressure, reroute steam, etc.). This flexibility allows us to explore conditions that would require significant infrastructure changes in a real-world setting at a fraction of the cost and time in a digital environment. The results highlight the benefits of using digital twins and advanced data analytics for the geothermal industry.

forecasting↗

DOE OSTI2022

Stochastic Learning Approach for Binary Optimization: Application to Bayesian Optimal Design of Experiments

Here, we present a novel stochastic approach to binary optimization suited for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The probabilistic objective is then solved by using a stochastic optimization routine to find an optimal observational policy. This formulation (a) is generally applicable to binary optimization problems with soft constraints and is ideal for OED and sensor placement problems; (b) does not require differentiability of the original objective function (e.g., a utility function in OED applications) with respect to the design variable, and thus it enables direct employment of sparsity-enforcing penalty functions such as $\ell_0$, without needing to utilize a continuation procedure or apply a rounding technique; (c) exhibits much lower computational cost than traditional gradient-based relaxation approaches; and (d) can be applied to both linear and nonlinear OED problems with proper choice of the utility function. The proposed approach is analyzed from an optimization perspective with detailed convergence analysis of the optimization approach and is also analyzed from a machine learning perspective with correspondence to policy gradient reinforcement learning. The approach is demonstrated numerically by using an idealized two-dimensional Bayesian linear inverse problem and validated by extensive numerical experiments carried out for sensor placement in a parameter identification setup.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2019

Everybody Needs Somebody Sometimes: Validation of Adaptive Recovery in Robotic Space Operations

This work assesses an adaptive approach to fault recovery in autonomous robotic space operations, which uses indicators of opportunity, such as physiological state measurements and observations of past human assistant performance, to inform future selections. We validated our reinforcement learning approach using data we collected from humans executing simulated mission scenarios. We present a method of structuring human-factors experiments that permits collection of relevant indicator of opportunity and assigned assistance task performance data, as well as evaluation of our adaptive approach, without requiring large numbers of test subjects. Application of our reinforcement learning algorithm to our experimental data shows that our adaptive assistant selection approach can achieve lower cumulative regret compared to existing non-adaptive baseline approaches when using real human data. Our work has applications beyond space robotics to any application where autonomy failures may occur that require external intervention.

Human-centered robotics↗

NASA NTRS2004

Product Distributions for Distributed Optimization

With connections to bounded rational game theory, information theory and statistical mechanics, Product Distribution (PD) theory provides a new framework for performing distributed optimization. Furthermore, PD theory extends and formalizes Collective Intelligence, thus connecting distributed optimization to distributed Reinforcement Learning (FU). This paper provides an overview of PD theory and details an algorithm for performing optimization derived from it. The approach is demonstrated on two unconstrained optimization problems, one with discrete variables and one with continuous variables. To highlight the connections between PD theory and distributed FU, the results are compared with those obtained using distributed reinforcement learning inspired optimization approaches. The inter-relationship of the techniques is discussed.

Bieniawski, Stefan R.↗