SEARCH · Search NASA

Results for “reward engineering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Discovering mechanisms for materials microstructure optimization via reinforcement learning of a generative model

Abstract The design of materials structure for optimizing functional properties and potentially, the discovery of novel behaviors is a keystone problem in materials science. In many cases microstructural models underpinning materials functionality are available and well understood. However, optimization of average properties via microstructural engineering often leads to combinatorically intractable problems. Here, we explore the use of the reinforcement learning (RL) for microstructure optimization targeting the discovery of the physical mechanisms behind enhanced functionalities. We illustrate that RL can provide insights into the mechanisms driving properties of interest in a 2D discrete Landau ferroelectrics simulator. Intriguingly, we find that non-trivial phenomena emerge if the rewards are assigned to favor physically impossible tasks, which we illustrate through rewarding RL agents to rotate polarization vectors to energetically unfavorable positions. We further find that strategies to induce polarization curl can be non-intuitive, based on analysis of learned agent policies. This study suggests that RL is a promising machine learning method for material design optimization tasks, and for better understanding the dynamics of microstructural simulations.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Electricity Pricing aware Deep Reinforcement Learning based Intelligent HVAC Control

Recently, deep reinforcement learning (DRL) based intelligent control of Heating, Ventilation, and Air Conditioning (HVAC) has gained a lot of attention due to DRL's ability to optimally control HVAC for minimizing operational cost while maintaining resident's comfort. The success of such DRL-based techniques largely depends on the articulation of the problem in terms of states, actions, and reward function. Inclusion of the electricity pricing information in the problem formulation can play an important role in saving the cost of HVAC operation. However, less attention has been given in the literature on formulating well-crafted state features based on electricity pricing. In this work, we propose an approach for training the DRL model with a specific focus on feature engineering based on electricity pricing. During training, we generate random but sufficiently realistic electricity price signals so that the pre-trained DRL model is robust and adaptive to the dynamic and variable electricity prices. The validation results are encouraging and show the potential of ≈12%-15% savings in the one day cost of HVAC operation, proving the usefulness of including electricity pricing related features as state features.

Kurte, Kuldeep↗

DOE OSTI2020

Single Episode Policy Transfer in Reinforcement Learning

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, we report our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

42 ENGINEERING↗

DOE OSTI2020

Incorporating uncertainty for enhanced leaderboard scoring and ranking in data competitions

Data competitions have become a popular and cost-effective approach for crowdsourcing versatile solutions from diverse expertise. Current practice relies on the simple leaderboard scoring based on a given set of competition data for ranking competitors and distributing the prize. However, a disadvantage of this practice in many competitions is that a slight difference in the scores due to the natural variability of the observed data could result in a much larger difference in the prize amounts. In this article, we propose a new strategy to quantify the uncertainty in the rankings and scores from using different data sets that share common characteristics with the provided competition data. By using a bootstrap approach to generate many comparable data sets, the new method has four advantages over current practice. Furthermore, during the competition, it provides a mechanism for competitors to get feedback about the uncertainty in their relative ranking. After the competition, it allows the host to gain a deeper understanding of the algorithm performance and their robustness across representative data sets. It also offers a transparent mechanism for prize distribution to reward the competitors more fairly with superior and robust performance. Finally, it has the additional advantage of being able to explore what results might have looked like if competition goals evolved from their original choices. The implementation of the strategy is illustrated with a real data competition hosted by Topcoder on urban radiation search.

42 ENGINEERING↗

DOE OSTI2024

Design, Selection, and Evaluation of Reinforcement Learning Single Agents for Ground Target Tracking

Previous approaches for small fixed-wing unmanned air systems that carry strapdown rather than gimbaled cameras achieved satisfactory ground target tracking performance using both standard and deep reinforcement learning algorithms. However, these approaches have significant restrictions and abstractions to the dynamics of the vehicle, such as constant airspeed and constant altitude, because the number of states and actions was necessarily limited. Thus, extensive tuning was required to obtain good tracking performance. The expansion from 4 state–action degrees of freedom to 15 enabled the agent to exploit previous reward functions that produced novel yet undesirable emergent behavior. This paper investigates the causes of and various potential solutions to undesirable emergent behavior in the ground target tracking problem. A combination of changes to the environment, reward structure, action space simplification, command rate, and controller implementation provides insight into obtaining stable tracking results. Consideration is given to reward structure selection and refinement to mitigate undesirable emergent behavior. Results presented in the paper for a simulated environment of a single unmanned air system tracking a randomly moving single ground target show that a soft actor–critic algorithm can produce feasible tracking trajectories without limiting the state space and action space, provided that the environment is properly posed.

Engineering↗

DOE OSTI2026

Financial-technical co-design for capital-intensive, resource-responsive energy systems

Because of their capital-intensive operation, wind energy systems that are competitive in terms of the cost of the energy that they produce lead to risk-reward trade-offs that make their business cases less favorable than those of conventional energy generation technologies. However, wind energy systems tend to be designed to maximize energy production or minimize cost of energy rather than to maximize their business cases. In this work, we attempt to exploit designs specifically tailored to business cases. We develop a novel framework for analyzing energy systems that ties their design variables to monthly operating incomes using simple models and historical hourly market and resource data. Using this approach, we demonstrate that for a wind site with abundant wind resource in the California Independent System Operator market, we can control the trade-off between mean and 5th percentile monthly returns by choosing the specific power of the turbine at a fixed modeled initial capital cost. Our framework gives a measure of the risk-reward spectrum of energy generation assets that could be built at a given site with respect to the sub-annual resource/market variation.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2021

DOE Energy Frontier Research Centers Center for Direct Catalytic Conversion of Biomass to Biofuels (C3Bio)

New capabilities to predict, design and control the chemistries of carbon could answer a global imperative to transition from fossil-based to sustainable transportation fuels. While the use of inexpensive hydrocarbons has been an unparalleled achievement and enabler of economic prosperity for many nations, singular dependence upon crude oil has given rise to systemic vulnerabilities in climate, energy, economic, and national security. Lignocellulosic biomass, a renewable and carbon-neutral resource, has the potential to displace an estimated annual equivalent of three billion barrels of oil in the U.S. alone (National Research Council 2009, U. S. Department of Energy, 2011). However, biomass has only one-third the energy density of crude oil (Agrawal and Singh 2009, Richard 2010) and lacks petroleum’s versatility as a feedstock for fuels and chemicals. These limitations keep biomass conversion below the efficiency level needed for strategic impact while the scientific challenge of routing carbon from one molecular context to another remains unmet. In 2009, the Center for Direct Catalytic Conversion of Biomass to Biofuels (C3Bio) recognized the potential of chemical catalysis and fast pyrolysis to overcome such limitations by transforming the main components of biomass (cellulose, xylan, and lignin) from grasses and trees directly to liquid hydrocarbons and aromatic co-products. Enabled by the EFRC high-risk, high-reward approach to grand challenge science, C3Bio researchers have been key national players in disrupting the conventional paradigm of the cellulosic biorefinery into a new future of “no carbon left behind”—the full utilization of carbon from plant cell walls in energy-dense fuels (Fig. 1). We identified catalytic and fast-pyrolytic pathways that utilize cellulose, xylan and, most significantly, lignin. We developed catalytic processes that deoxygenate and transform monomers and isolated polymers into useful products and tested their use with intact biomass. We gained control of lignin synthesis within plants and initiated tailoring biomass to its end-use through the tools of plant molecular biology and genetic engineering. C3Bio breakthroughs have increased the energy density of biomass-derived substrates via catalytic and pyrolytic conversions into products such as benzoquinones, furfural and hydroxymethylfurfural, levoglucosan and levulinic acid, methoxypropylphenols and propylbenzene. Such advances in biomass conversion would not have been possible without simultaneous advances in analytical instrumentation and methodologies, and imaging technologies and applications. The legacy science developed by C3Bio enables design and control strategies for achieving a targeted product portfolio of fuel and chemical feedstocks from a diverse range of native and tailored biomass. Our research provides the knowledge base required for a bio-economy with product streams as diverse in functionality as those of the petrochemical industry. Coupling targeted computational modeling with experimentation, we achieved: (1) fundamental understanding of biopolymers and cell wall architecture assembly, (2) discovery of new chemistries that allow the development of highly selective pathways to fuels and desirable chemicals, and (3) an integrated systems-level understanding to control catalytic and pyrolytic pathways.

09 BIOMASS FUELS↗

DOE OSTI2022

Variational multiscale reinforcement learning for discovering reduced order closure models of nonlinear spatiotemporal transport systems

Abstract A central challenge in the computational modeling and simulation of a multitude of science applications is to achieve robust and accurate closures for their coarse-grained representations due to underlying highly nonlinear multiscale interactions. These closure models are common in many nonlinear spatiotemporal systems to account for losses due to reduced order representations, including many transport phenomena in fluids. Previous data-driven closure modeling efforts have mostly focused on supervised learning approaches using high fidelity simulation data. On the other hand, reinforcement learning (RL) is a powerful yet relatively uncharted method in spatiotemporally extended systems. In this study, we put forth a modular dynamic closure modeling and discovery framework to stabilize the Galerkin projection based reduced order models that may arise in many nonlinear spatiotemporal dynamical systems with quadratic nonlinearity. However, a key element in creating a robust RL agent is to introduce a feasible reward function, which can be constituted of any difference metrics between the RL model and high fidelity simulation data. First, we introduce a multi-modal RL to discover mode-dependant closure policies that utilize the high fidelity data in rewarding our RL agent. We then formulate a variational multiscale RL (VMRL) approach to discover closure models without requiring access to the high fidelity data in designing the reward function. Specifically, our chief innovation is to leverage variational multiscale formalism to quantify the difference between modal interactions in Galerkin systems. Our results in simulating the viscous Burgers equation indicate that the proposed VMRL method leads to robust and accurate closure parameterizations, and it may potentially be used to discover scale-aware closure models for complex dynamical systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Giant g-factors and fully spin-polarized states in metamorphic short-period InAsSb/InSb superlattices

Abstract Realizing a large Landé g -factor of electrons in solid-state materials has long been thought of as a rewarding task as it can trigger abundant immediate applications in spintronics and quantum computing. Here, by using metamorphic InAsSb/InSb superlattices (SLs), we demonstrate an unprecedented high value of g ≈ 104, twice larger than that in bulk InSb, and fully spin-polarized states at low magnetic fields. In addition, we show that the g -factor can be tuned on demand from 20 to 110 via varying the SL period. The key ingredients of such a wide tunability are the wavefunction mixing and overlap between the electron and hole states, which have drawn little attention in prior studies. Our work not only establishes metamorphic InAsSb/InSb as a promising and competitive material platform for future quantum devices but also provides a new route toward g -factor engineering in semiconductor structures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2022

Game-Theoretic Strategies for Cyber-Physical Infrastructures Under Component Disruptions

In this work, networked infrastructures of recursively defined systems composed of discrete cyber and physical components are considered. The components of basic systems at the finest levels can be disrupted by cyber or physical means, and can be reinforced to survive at certain costs. A problem of ensuring the infrastructure performance is formulated as a game between a provider and an attacker, who probabilistically choose components to reinforce and attack, respectively. The disruptions of this infrastructure are characterized using the aggregate failure correlation function that specifies the conditional failure probability of the infrastructure given the failure of an individual system at that level. The survival probabilities of basic systems satisfy simple product-form, first-order differential equations expressed in terms of the multiplier functions. The utility functions of the provider and attacker are composed of the reward and cost terms, both expressed in terms of the component reinforcement and attack probabilities. The Nash equilibrium of this game is characterized, along with the sensitivity functions of the survival probabilities of basic systems that highlight their dependence individually on the cost-benefit terms, the correlation functions, and the multiplier functions. These results are illustrated using simplified models of a distributed cloud servers infrastructure, a 5G data network infrastructure, a high performance computing federation, and a smart energy grid infrastructure.

42 ENGINEERING↗

DOE OSTI2022

Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner

We report in this paper describes how the performance of motion primitive-based planning algorithms can be improved using reinforcement learning. Specifically, we describe and evaluate a framework that autonomously improves the performance of a primitive-based motion planner. The improvement process consists of three phases: exploration, extraction, and reward updates. This process can be iterated continuously to provide successive improvement. The exploration step generates new trajectories, and the extraction step identifies new primitives from these trajectories. These primitives are then used to update rewards for continued exploration. This framework required novel shaping rewards, development of a primitive extraction algorithm, and modification of the Hybrid A* algorithm. The framework is tested on a navigation task using a nonlinear F-16 model. The framework autonomously added 91 motion primitives to the primitive library and reduced average path cost by 21.6 seconds, or 35.75% of the original cost. The learned primitives are applied to an obstacle field navigation task, which was not used in training, and reduced path cost by 16.3 seconds, or 24.1%. Additionally, two heuristics for the modified Hybrid A* algorithm are designed to improve effective branching factor.

42 ENGINEERING↗

DOE OSTI2022

Hypothesis testing via AI: Generating physically interpretable models of scientific data with machine learning (Full Technical Report)

Deep learning has demonstrated an exceptional ability to solve complex tasks (an engineering success); however, it has done so at the expense of the ability to generate new knowledge (a scientific failure). We propose an alternative framework—entitled Deep Symbolic Regression (DSR)—in which artificial neural networks (NNs) rapidly generate hypotheses about physical relationships among inputs. This framework bypasses the need to interpret an NN altogether, while still leveraging the representational power of deep learning. The resulting models are tractable mathematical expressions, which are inherently and readily human interpretable and can provide insights into underlying physical phenomena. Further, we fold this methodology into the scientific process by allowing the scientist to directly integrate a priori knowledge and beliefs to accelerate learning. We demonstrate this methodology on symbolic regression—the problem of rediscovering underlying expressions describing a dataset—and achieve state-of-the-art performance across a wide variety of symbolic regression problems. Further, we generalize our DSR framework to apply to the more general class of symbolic optimization problems, in which one seeks to optimize a sequence of symbols or “tokens” under a black-box reward function. Examples of other symbolic optimization problems include neural architecture search and computational antibody design. Our generalized tool, Deep Symbolic Optimization (DSO), has been demonstrated on the task of learning symbolic control policies for reinforcement learning environments, and has been adopted as an enabling capability for computational antibody design.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Investing in STEM Benefits Community, Company, and Industry - 20243

Like many organizations and groups, the Department of Energy Environmental Management program and the nuclear industry are facing an aging employee population and record numbers of employee retirements. This 'silver tsunami' of retirees means the industry as a whole is recruiting graduates at a fast pace. So, how do we ensure the nuclear industry has a robust pipeline of graduates with the necessary background? Fluor Idaho's investment in science, technology, engineering, and mathematics (STEM) education programs helps promote the next generation of technical professionals, benefiting both the company and the industry. Through community outreach initiatives and student mentoring programs, Fluor Idaho promotes kids' interest in STEM. The programs are designed to spark STEM interest in grade school and then foster and grow that interest through high school and higher education. After igniting student excitement about STEM topics in school, Fluor Idaho focuses on providing work-specific opportunities including mentoring and intern programs. These programs give students a better look at real-world job opportunities while still stimulating their interest in STEM. These programs ultimately provide career opportunities for the students and a pipeline of workers. Fluor Idaho's strategy for its summer interns has been to place students in a wide variety of projects - from the commissioning of a first-of-a-kind nuclear processing facility, to operating nuclear waste management facilities, to developing computer programs and applications - and give them responsibilities that make them feel like valued contributors. This approach has created a positive and rewarding summer-intern program experience for the students. Fluor Idaho's investment in its future today is reducing the knowledge gap - or cliff - that many contractors in the nuclear industry are facing. It's a worthwhile investment that is readily visible within its many cleanup projects. The next generation of technical professionals is making an impact now - benefiting both the company and the nuclear industry. This paper will review Fluor Idaho's programs, including STEM community outreach and education and mentoring programs. It will also present statistics about the reach of those programs and their successes. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2023

Reinforcement learning-based design of shape-changing metamaterials

During the last decade, artificially architected materials have been designed to obtain properties unreachable by naturally occurring materials, whose properties are determined by their atomic structure and chemical composition. In this work, we implement a new reinforcement learning (RL) method able to rationally design unique metamaterial structures at the nano-, micro-, and macroscale, which change shape during operational conditions. As an example, we apply this method to design nanostructured silicon anodes for Li-ion batteries (LIBs). The RL model is designed to apply different actions and predict change during operational conditions. The multi-component reward function comprises an increase in the total storage capacity of the resulting battery electrode and structural parameters, such as the minimum distance between the individual components of the nanostructure. Upon experimental validation using a polymer-based 3D printing technique, we expect that the newly discovered structures improve the current Si-based LIB anodes state-of-the-art by almost three times and almost ten times the current commercial LIB based on a graphitic anode. Furthermore, this RL-based optimization method opens up vast design space for other responsive metamaterials with tailored properties and pre-programmed structural transformation.

25 ENERGY STORAGE↗

DOE OSTI2024

Challenges and Solutions for Leave-One-Out Biosensor Design in the Context of a Rugged Fitness Landscape

The leave-one-out (LOO) green fluorescent protein (GFP) approach to biosensor design combines computational protein design with split protein reconstitution. LOO-GFPs reversibly fold and gain fluorescence upon encountering the target peptide, which can be redefined by computational design of the LOO site. Such an approach can be used to create reusable biosensors for the early detection of emerging biological threats. Enlightening biophysical inferences for nine LOO-GFP biosensor libraries are presented, with target sequences from dengue, influenza, or HIV, replacing beta strands 7, 8, or 11. An initially low hit rate was traced to components of the energy function, manifesting in the over-rewarding of over-tight side chain packing. Also, screening by colony picking required a low library complexity, but designing a biosensor against a peptide of at least 12 residues requires a high-complexity library. This double-bind was solved using a “piecemeal” iterative design strategy. Also, designed LOO-GFPs fluoresced in the unbound state due to unwanted dimerization, but this was solved by fusing a fully functional prototype LOO-GFP to a fiber-forming protein, Drosophila ultrabithorax, creating a biosensor fiber. One influenza hemagglutinin biosensor is characterized here in detail, showing a shifted excitation/emission spectrum, a micromolar affinity for the target peptide, and an unexpected photo-switching ability.

Chemistry↗

DOE OSTI2025

Closing the Loop between In Situ Stress Complexity and EGS Fracture Complexity

We present an agent-guided approach to CAD geometry decomposition that automates hex/hybrid meshing with graph neural networks (GNNs) to accelerate next-generation ModSim workflows. Our end-to-end pipeline (i) reduces 3D boundary-representation (B-Rep) models to a 2D chordal axis skeleton (CAT) and then to a 1D bipartite graph of surface and curve nodes, (ii) assigns per node labels as Cubit® WebCut actions, (iii) trains a multi-action GNN under supervised learning, and (iv) predicts five surface-node and three curve-node actions on out-of-distribution test geometries. Each graph node carries geometric, topological, and meshing attributes drawn from the B-Rep “skin” and CAT “skeleton,” with two-way mappings across 3D↔2D↔1D representations to maintain traceability back to 3D CAD. The supervised learning model exhibits stable convergence of the binary cross-entropy loss and achieves 98.7% accuracy on unseen lattice models. To operationalize decision-making, we rank predicted commands by geometric significance and prototyped the agent-guided workflow through the Cubit® Meshing PowerTool GUI. As a stretch goal, we explore reinforcement learning (RL) to reduce or remove label requirements and to learn policies for action sequences that maximize total reward (e.g., size of hex-meshable regions and resulting hex mesh quality). When all-hex meshing is not feasible, the agent assists in producing hybrid meshes—prioritizing hex in critical regions and transitioning to tetrahedral elements (tets) elsewhere—maintaining fidelity while ensuring robustness. The overarching objective is to replace manual, heuristics-based decomposition with data-driven, reproducible automation, cutting meshing turnaround time by orders of magnitude. We anticipate direct impact on simulation workflows through intelligent, scalable decomposition of complex CAD models into hex-meshable subdomains.

42 ENGINEERING↗

DOE OSTI2022

Reinforcement learning for online adaptation of model predictive controllers: Application to a selective catalytic reduction unit

Here we present a novel application of reinforcement learning (RL) for online dynamic tuning of model predictive controllers (MPC). Applying a state-action-reward-state-action (SARSA) algorithm for temporal difference learning with a control-specific reward function improves the error tracking performance of a standard MPC formulation. The proposed RL approach is also readily adaptable to other MPCs, or entirely different control approaches. Practical details for the implementation of the RL-MPC algorithm are also presented. The proposed algorithm is applied to a case study of controlling nitrogen oxide (NO x ) emissions in an industrial selective catalytic reduction (SCR) unit, a control problem characterized by significant nonlinearity and time delay. Along with an RL-MPC formulation for NOx control, another MPC is proposed to mitigate ammonia slip and decrease ammonia consumption in the SCR. Results showing the efficacy of the RL-MPC for NO x control through learning and implementation on the nonlinear SCR dynamic model are presented.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2025

Topology-Aware Reinforcement Learning for Voltage Control: Centralized and Decentralized Strategies

Volt-VAR control (VVC) methods based on deep reinforcement learning (DRL) can effectively control distribution grid voltage and minimize power loss by implementing corrective and preventive control measures on the reactive power output of inverter-based distributed energy resources (DERs). However, model-free DRL-based VVC approaches usually cannot capture the important topological feature of the power system since they use a fully-connected network (FCN) to deliver the action. Therefore, this paper proposes a graph convolutional network (GCN)-based DRL approach that can employ the topological information of the network to take better control action for regulating the voltage. Our implementation allows for both centralized and decentralized configurations, utilizing a single agent and multiple agents respectively. Although the centralized GCN-based DRL approach has its advantages of minimizing voltage fluctuation and power loss, it is not suitable for large scale power systems due to its challenges in terms of scalability, computation speed and potential single points of failure. Therefore, these problems can be resolved using the decentralized GCN-based DRL approach. Moreover, to ensure the safe operation of the model, our proposed approach incorporates an exponential barrier function while formulating the reward function for each agent. To validate performance of the proposed approaches, the proposed model is tested on modified IEEE test systems and the performances are measured in terms on voltage fluctuation reduction, minimization of power loss and computational speed. Finally, the results show that the proposed topology-aware approach outperforms the FCN-based DRL approach in terms of reducing voltage fluctuation and minimizing power loss of the network. Moreover, it is shown that the decentralized GCN-based DRL has faster computational speed than other approaches.

42 ENGINEERING↗