Search NASA⌕ Search

SEARCH · Search NASA

Results for “inverse reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

31 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Learning with Delayed Rewards—A Case Study on Inverse Defect Design in 2D Materials

Defect dynamics in materials are of central importance to a broad range of technologies from catalysis to energy storage systems to microelectronics. Material functionality depends strongly on the nature and organization of defects-their arrangements often involve intermediate or transient states that present a high barrier for transformation. The lack of knowledge of these intermediate states and the presence of this energy barrier presents a serious challenge for inverse defect design, especially for gradient-based approaches. Here, we present a reinforcement learning (RL) [Monte Carlo Tree Search (MCTS)] based on delayed rewards that allow for efficient search of the defect configurational space and allows us to identify optimal defect arrangements in low-dimensional materials. Using a representative case of two-dimensional MoS 2 , we demonstrate that the use of delayed rewards allows us to efficiently sample the defect configurational space and overcome the energy barrier for a wide range of defect concentrations (from 1.5 to 8% S vacancies)-the system evolves from an initial randomly distributed S vacancies to one with extended S line defects consistent with previous experimental studies. Detailed analysis in the feature space allows us to identify the optimal pathways for this defect transformation and arrangement. Comparison with other global optimization schemes like genetic algorithms suggests that the MCTS with delayed rewards takes fewer evaluations and arrives at a better quality of the solution. The implications of the various sampled defect configurations on the 2H to 1T phase transitions in MoS 2 are discussed. In this study, we introduce a RL strategy employing delayed rewards that can accelerate the inverse design of defects in materials for achieving targeted functionality.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Reinforcement learning based automated history matching for improved hydrocarbon production forecast

History matching aims to find a numerical reservoir model that can be used to predict the reservoir performance. An engineer and model calibration (data inversion) method are required to adjust various parameters/properties of the numerical model in order to match the reservoir production history. In this study, we develop deep neural networks within the reinforcement learning framework to achieve automated history matching that will reduce engineers’ efforts, human bias, automatically and intelligently explore the parameter space, and remove the need of large set of labeled training data. To that end, a fast-marching-based reservoir simulator is encapsulated as an environment for the proposed reinforcement learning. The deep neural-network-based learning agent interacts with the reservoir simulator within reinforcement learning framework to achieve the automated history matching. Reinforcement learning techniques, such as discrete Deep Q Network and continuous Deep Deterministic Policy Gradients, are used toth, used to train the learning agents. The continuous actions enable the Deep Deterministic Policy Gradients to explore more states at each iteration in a a learning episode; consequently, a better history matching is achieved using this algorithm as compared to Deep Q Network. For simplified dual-target composite reservoir models, the best history-matching performances of the discrete and continuous learning methods in terms of normalized root mean square errors are 0.0447 and 0.0038, respectively. Furthermore, our study shows that continuous action space achieved by the deep deterministic policy gradient drastically outperforms deep Q network.

42 ENGINEERING↗
DOE OSTI2026

Artificial intelligence-driven approaches for materials design and discovery

Materials design is an important component of modern science and technology, yet traditional approaches rely heavily on trial and error and can be inefficient. Computational techniques, enhanced by modern artificial intelligence, have reshaped the landscape of designing new materials. Among these approaches, inverse design has shown great promise in designing materials that meet specific property requirements. Here, in this Review, we present key computational advances in materials design over the past few decades. We follow the evolution of relevant materials design techniques, from high-throughput forward machine learning methods and evolutionary algorithms, to advanced artificial intelligence strategies such as reinforcement learning and deep generative models. We highlight the paradigm shift from conventional screening approaches to inverse generation driven by deep generative models. Finally, we discuss current challenges and future perspectives of materials inverse design. This Review may serve as a brief guide to the approaches, progress and outlook of designing future functional materials with technological relevance.

computational methods↗
DOE OSTI2022

End-to-end optimization for battery materials and molecules by combining graph neural networks and reinforcement learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to design new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗
DOE OSTI2022

End-to-End Optimization for Battery Materials and Molecules by Combining Graph Neural Networks and Reinforcement Learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to the design of new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗
DOE OSTI2022

A machine learning approach to determine the elastic properties of printed fiber-reinforced polymers

This work focuses on the simultaneous determination of the elastic constants and the fiber orientation state for a short fiber-reinforced polymer composite by performing a minimum of experimental tests. Here we introduce a methodology that enables the inverse determination of fiber orientation state and the in-situ polymer properties by performing tensile tests at the composite coupon level. We demonstrate the approach for the extrusion deposition additive manufacturing (EDAM) process to illustrate one application of the methodology, but the development is such that it can be applied to short fiber-reinforced polymer (SFRP) systems processed via other methods. Currently, developing composites additive manufacturing digital twins require extensive material characterization. In particular, the mechanical characterization of the orthotropic elastic properties of a composite involves extensive sample preparation and testing, therefore the elasticity tensor is generally populated using a micromechanics model. This, however, requires measuring the fiber orientation state in addition to knowing the constituent material properties. Experimentally measuring the fiber orientation state can be tedious and time consuming. Further, optical methods are limited to resolving the orientation of cylindrical fibers or cluster of non-cylindrical fibers, and computed tomography (CT) methods scan regions of volume that are much smaller than a full printed bead. Therefore, we propose a methodology, accelerated by machine learning, to identify the anisotropic mechanical properties and fiber orientation state at the same time. Early results show that inference of the fiber orientation and composite properties is possible with as few as three tensile tests. Our results show that a combination of the choice of the micromechanics model and reliable set of experiments can yield the nine elastic constants, as well as, the fiber orientation state.

36 MATERIALS SCIENCE↗
NASA NTRS2024

Harnessing Collaborative Learning Automata to Guide Multi-objective Optimization based Inverse Analysis for Structural Damage Identification

Structural damage identification based on physical models is often transformed into an optimization problem that minimizes the difference between measurement information of structure being monitored and the model prediction in the parametric space. However, the objective function in this context often exhibits multimodality, involving high-dimensional variables due to the reliance on finite element models for damage identification. These features pose challenges to optimization algorithms, where entrapment in local solutions can lead to false positives and false negatives in damage identification. In this research, we propose a reinforcement learning based multi-swarm optimizer to tackle such challenges in pursuit of a small yet diverse solution set that can capture the true damage scenario as one of the solutions. The proposed method leverages the flexibility of the particle swarm optimizer and incorporates novel strategies of metaheuristics to realize targeted improvement. To enable the particle swarm to adaptively select the appropriate search strategy based on the current environment, we adopt the learning automata technique, which sidesteps the need for reward strategy selection that is usually ad hoc at each step of the search. The integration harnesses the automatic learning and self-adaptation capabilities of learning automata, enabling the particles to navigate based on environmental signals. This leads to accumulated probabilities tied to advantageous movements, fostering an adaptive exploration of particles in the search space. The proposed approach is first validated through implementing into benchmark test cases with comparisons. It is then applied to structural damage identification with piezoelectric admittance experimental signals. `The results highlight the capability of the algorithm to identify a small solution set with high accuracy to match the actual damage scenario.

Yang Zhang↗
NASA NTRS2024

Piezoelectric impedance-based high-accuracy damage identification using sparsity conscious multi-objective optimization inverse analysis

Two elements are essential in structural health monitoring utilizing dynamic responses: response measurement with high-frequency contents, i.e., small characteristic wavelengths, that can adequately reflect damage features, and effective inverse identification analysis that is however oftentimes under-determined. The advancement of smart structure integration has led to active interrogation through frequency-sweeping piezoelectric impedance measurement at high frequency range. In this research we develop a multi-objective optimization formulation for the identification of damage location and severity utilizing piezoelectric impedance. While one optimization objective is to match the response measurement with finite element model prediction in the damage parametric space, the other is the number of locations of damage, i.e., the sparsity of damage index as the solution vector, since damage usually occurs within a small number of locations. This multi-objective formulation fits well the under-determined nature of damage identification, as it naturally provides multiple solutions as basis for further elucidation. The challenge remaining is how to find a small solution set that can include the actual damage scenario. Here we develop a novel inverse identification framework utilizing the intelligent swarm optimizer which possesses flexibility for enhancement. We first embed a sparsity enforcement process into the population generation of the optimizer, which yields a solution repository intrinsically possessing sparsity. We then apply reinforcement learning so the agents can adaptively opt for local strategies with the aim of enriching the searching patterns to diversify the solutions. Through the incorporation of a Q-table, searching toward more promising directions will be rewarded. Our case analyses employing experimental data indicate that this sparsity-conscious multi-objective particle swarm optimization technique can lead to a small solution set which generally encompasses the true damage scenario. This effectively solves the structural damage identification problem with piezoelectric impedance measurement.

Yang Zhang↗
DOE OSTI2023

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor-critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection-diffusion field. As a result, we provide explanation for the policy behaviors using knowledge of the underlying physical process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Grand canonical inverse design of multicomponent colloidal crystals

Inverse design methods are powerful computational approaches for creating colloidal systems which self-assemble into a target morphology by reverse engineering the Hamiltonian of the system. Despite this, these optimization procedures tend to yield Hamiltonians which are too complex to be experimentally realized. An alternative route to complex structures involves the use of several different components, however, conventional inverse design methods do not explicitly account for the possibility of phase separation into compositionally distinct structures. We present an inverse design scheme for multicomponent colloidal systems by combining active learning with a method to directly compute their ground state phase diagrams. This explicitly accounts for phase separation and can locate stable regions of Hamiltonian parameter space which grid-based surveys are prone to miss. Using this we design low-density, binary structures with Lennard-Jones-like pairwise interactions that are simpler than in the single component case and potentially realizable in an experimental setting. This reinforces the concept that ground states of simple, multicomponent systems might be rich with previously unappreciated diversity, enabling the assembly of non-trivial structures with only few simple components instead of a single complex one.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Machine learning-driven design and self-sensing capabilities of automotive bumper lattices for adaptive impact response

We present a novel approach to design an automotive bumper energy absorber using carbon fiber reinforced polymer composites, optimized to meet conflicting performance requirements for two distinct impact scenarios. The design must satisfy both a low-speed (2.5 mph) pendulum intrusion test, simulating vehicle-to-vehicle collisions, and a high-speed (25 mph) leg flexion test, replicating pedestrian impacts. These tests demand opposing deformation characteristics: high flexibility (deformation < 85 mm) for the former and high stiffness (deformation < 22 mm) for the latter. To address these contradictory requirements, we developed a machine learning (ML) framework for inverse optimization of lattice designs and material selection. Unlike traditional iterative design processes, our ML model directly outputs optimal design parameters and material choices based on target performance inputs. The energy absorber was fabricated using advanced additive manufacturing techniques, including extrusion deposition and digital light processing. The integration of carbon fibers provides multifunctionality to the bumper structure, enabling self-sensing capabilities through changes in electrical resistivity under compression. This electrical response demonstrates high repeatability under multiple cycles at 2% compression and exhibits distinct signatures during crack formation under high deformation. This research offers adaptive performance through innovative design methodologies and smart material integration. The approach has potential applications in various fields requiring adaptive energy absorption and real-time structural health monitoring.

Chawla, Komal [ORNL] (ORCID:0000000190327565)↗
DOE OSTI2025

Physically constrained 3D diffusion for inverse design of fiber-reinforced polymer composite materials

Designing fiber-reinforced polymer composites (FRPCs) with a tailored nonlinear stress-strain response is crucial for applications such as energy absorption in crash structures, flexible robotics, and impact-resistant protective gear. However, the inherent complexities of composite materials and the multitude of parameters involved, render traditional design and optimization methods inadequate for achieving effective inverse design of composites. In this paper, we present an AI-based inverse design framework that effectively and efficiently generates FRPCs with targeted nonlinear stress-strain responses. We introduce a physically constrained diffusion model (PC3D_Diffusion) capable of managing the complexities of composite materials and producing detailed, high-quality designs. We propose a loss-guided, learning-free approach to generate physically feasible microstructure designs by explicitly enforcing physical constraints during the generation process. For training purposes, 1.35 million FRPC samples were created, and their corresponding stress-strain curves were computed using established physics-based computational models. The results show that PC3D_Diffusion consistently generates high-quality designs with tailored mechanical behaviors, while guaranteeing compliance with the physical constraints. PC3D_Diffusion advances FRPC inverse design and may facilitate the inverse design of other 3D materials, offering potential applications in industries reliant on materials with custom mechanical properties.

Xu, Pei [Clemson Univ., SC (United States)]↗
DOE OSTI2024

Ab Initio-Based Bond Order Potential for Arsenene Polymorphs Developed via Hierarchical Reinforcement Learning

Arsenene, a less-explored two-dimensional material, holds the potential for applications in wearable electronics, memory devices, and quantum systems. This study introduces a bond-order potential model with Tersoff formalism, the ML-Tersoff, which leverages multireward hierarchical reinforcement learning (RL), trained on an ab initio data set. This data set covers a spectrum of properties for arsenene polymorphs, enhancing our understanding of its mechanical and thermal behaviors without the complexities of traditional models requiring multiple parameter sets. Our RL strategy utilizes decision trees coupled with a hierarchical reward strategy to accelerate convergence in high-dimensional continuous search spaces. Unlike the Stillinger-Weber approach, which demands separate formalisms for buckled and puckered forms, the ML-Tersoff model concurrently captures multiple properties of the two polymorphs by effectively representing the local environment, thereby avoiding the need for different atomic types. Here, we apply the ML model to understand the mechanical and thermal properties of the arsenene polymorphs and nanostructures. We observe an inverse relationship between the critical strain and temperature in arsenene. Thermal conductivity calculations in nanosheets show good agreement with ab initio data, reflecting a decrease in thermal conductivity attributable to increased anharmonic effects at higher temperatures. We also apply the model to predict the thermal behavior of arsenene nanotubes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗