Search NASA⌕ Search

SEARCH · Search NASA

Results for “reward engineering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

aphBO-2GP-3B: a budgeted asynchronous parallel multi-acquisition functions for constrained Bayesian optimization on high-performing computing architecture

High-fidelity complex engineering simulations are often predictive, but also computationally expensive and often require substantial computational efforts. The mitigation of computational burden is usually enabled through parallelism in high-performance cluster (HPC) architecture. Optimization problems associated with these applications is a challenging problem due to the high computational cost of the high-fidelity simulations. In this paper, an asynchronous parallel constrained Bayesian optimization method is proposed to efficiently solve the computationally expensive simulation-based optimization problems on the HPC platform, with a budgeted computational resource, where the maximum number of simulations is a constant. The advantage of this method are three-fold. Firstly, the efficiency of the Bayesian optimization is improved, where multiple input locations are evaluated parallel in an asynchronous manner to accelerate the optimization convergence with respect to physical runtime. This efficiency feature is further improved so that when each of the inputs is finished, another input is queried without waiting for the whole batch to complete. Second, the proposed method can handle both known and unknown constraints. Third, the proposed method samples several acquisition functions based on their rewards using a modified GP-Hedge scheme. The proposed framework is termed aphBO-2GP-3B, which means asynchronous parallel hedge Bayesian optimization with two Gaussian processes and three batches. The numerical performance of the proposed framework aphBO-2GP-3B is comprehensively benchmarked using 16 numerical examples, compared against other 6 parallel Bayesian optimization variants and 1 parallel Monte Carlo as a baseline, and demonstrated using two real-world high-fidelity expensive industrial applications. The first engineering application is based on finite element analysis (FEA) and the second one is based on computational fluid dynamics (CFD) simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

IMPROVING TRANSPORTATION EFFICIENCY THROUGH INTEGRATED VEHICLE, ENGINE, AND POWERTRAIN RESEARCH -SUPERTRUCK II

Daimler Truck North America (DTNA) completed a five year, $40.1 M SuperTruck 2 project to demonstrate technologies to achieve both vehicle and engine efficiency improvements. The vehicle objective was to develop and demonstrate a concept vehicle with at least 115% vehicle freight efficiency improvement over a weighted average of four cycles relative to a 2009 best-in-class baseline vehicle. The engine technology development goal was to achieve 55% brake thermal efficiency (BTE) as tested on a dynamometer at an equivalent of 65 MPH. Both project objectives were to develop technologies that are cost effective. SuperTruck 2 started in 2017 and built on the knowledge gained in SuperTruck 1. SuperTruck 2 enabled the ability to evaluate high risk high reward research centered on four areas that show the most potential for commercialization: aerodynamics, powertrain, rolling resistance and energy management. In addition, several industry, University and National laboratories were able to collaborate in developing technologies found to be successful in meeting the program objectives. Several technologies show promise towards production while some technologies do not show a quick path towards production.

Villeneuve, Darek↗
NASA NTRS2014

Microwave Interferometric Radiometry in Remote Sensing: an Invited Historical Review

The launch of the Soil Moisture and Ocean Salinity (SMOS) mission on 2 November 2009 marked a milestone in remote sensing for it was the first time a radiometer capable of acquiring wide field of view images at every single snapshot, a unique feature of the synthetic aperture technique, made it to space. The technology behind such an achievement was developed, thanks to the effort of a community of researchers and engineers in different groups around the world. It was only because of their joint work that SMOS finally became a reality. The fact that the European Space Agency, together with CNES (Centre National d'Etudes Spatiales) and CDTI (Centro para el Desarrollo Tecnológico e Industrial), managed to get the project through should be considered a merit and a reward for that entire community. This paper is an invited historical review that, within a very limited number of pages, tries to provide insight into some of the developments which, one way or another, are imprinted in the name of SMOS.

Microwave↗
DOE OSTI2020

Optimal carbon storage reservoir management through deep reinforcement learning

Model-based optimization plays a central role in energy system design and management. The complexity and high-dimensionality of many process-level models, especially those used for geosystem energy exploration and utilization, often lead to formidable computational costs when the dimension of decision space is also large. This work adopts elements of recently advanced deep learning techniques to solve a sequential decision-making problem in applied geosystem management. Specifically, a deep reinforcement learning framework was formed for optimal multiperiod planning, in which a deep Q-learning network (DQN) agent was trained to maximize rewards by learning from high-dimensional inputs and from exploitation of its past experiences. To expedite computation, deep multitask learning was used to approximate high-dimensional, multistate transition functions. Both DQN and deep multitask learning are pattern based. As a demonstration, the framework was applied to optimal carbon sequestration reservoir planning using two different types of management strategies: monitoring only and brine extraction. Both strategies are designed to mitigate potential risks due to pressure buildup. Results show that the DQN agent can identify the optimal policies to maximize the reward for given risk and cost constraints. Finally, experiments also show that knowledge the agent gained from interacting with one environment is largely preserved when deploying the same agent in other similar environments.

15 GEOTHERMAL ENERGY↗
DOE OSTI2022

Discovering mechanisms for materials microstructure optimization via reinforcement learning of a generative model

Abstract The design of materials structure for optimizing functional properties and potentially, the discovery of novel behaviors is a keystone problem in materials science. In many cases microstructural models underpinning materials functionality are available and well understood. However, optimization of average properties via microstructural engineering often leads to combinatorically intractable problems. Here, we explore the use of the reinforcement learning (RL) for microstructure optimization targeting the discovery of the physical mechanisms behind enhanced functionalities. We illustrate that RL can provide insights into the mechanisms driving properties of interest in a 2D discrete Landau ferroelectrics simulator. Intriguingly, we find that non-trivial phenomena emerge if the rewards are assigned to favor physically impossible tasks, which we illustrate through rewarding RL agents to rotate polarization vectors to energetically unfavorable positions. We further find that strategies to induce polarization curl can be non-intuitive, based on analysis of learned agent policies. This study suggests that RL is a promising machine learning method for material design optimization tasks, and for better understanding the dynamics of microstructural simulations.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Electricity Pricing aware Deep Reinforcement Learning based Intelligent HVAC Control

Recently, deep reinforcement learning (DRL) based intelligent control of Heating, Ventilation, and Air Conditioning (HVAC) has gained a lot of attention due to DRL's ability to optimally control HVAC for minimizing operational cost while maintaining resident's comfort. The success of such DRL-based techniques largely depends on the articulation of the problem in terms of states, actions, and reward function. Inclusion of the electricity pricing information in the problem formulation can play an important role in saving the cost of HVAC operation. However, less attention has been given in the literature on formulating well-crafted state features based on electricity pricing. In this work, we propose an approach for training the DRL model with a specific focus on feature engineering based on electricity pricing. During training, we generate random but sufficiently realistic electricity price signals so that the pre-trained DRL model is robust and adaptive to the dynamic and variable electricity prices. The validation results are encouraging and show the potential of ≈12%-15% savings in the one day cost of HVAC operation, proving the usefulness of including electricity pricing related features as state features.

Kurte, Kuldeep↗
NASA NTRS1972

What motivates researchers in times of economic uncertainty.

Results of a study initiated late in 1970 to obtain both a measure of on-and-around-the-job factors which were 'motivating' to engineers and scientists, and to obtain an indication of how the relative importance of these factors changes as a result of the uncertain economic environment. A questionnaire, 'The Jackman Job Satisfaction Schedule,' was used to satisfy the needs of the study. It is concluded that managers can enhance the feeling of motivation by making individual job assignments interesting and challenging, by formulating significant milestones and end points into job content, and by assigning ample rewards with corresponding responsibility. In times of economic uncertainty increased emphasis should be given to security-related aspects of employment.

Bucher, G. C.↗
DOE OSTI2020

Single Episode Policy Transfer in Reinforcement Learning

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, we report our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

42 ENGINEERING↗
NASA NTRS1999

A Survey of Collective Intelligence

This chapter presents the science of "COllective INtelligence" (COIN). A COIN is a large multi-agent systems where: i) the agents each run reinforcement learning (RL) algorithms; ii) there is little to no centralized communication or control; iii) there is a provided world utility function that, rates the possible histories of tile full system. Tile conventional approach to designing large distributed systems to optimize a world utility does not use agents running RL algorithms. Rather that approach begins with explicit modeling of the overall system's dynamics, followed by detailed hand-tuning of the interactions between the components to ensure that they "cooperate" as far as the world utility is concerned. This approach is labor-intensive, often results in highly non-robust systems, and usually results in design techniques that, have limited applicability. In contrast, with COINs we wish to solve the system design problems implicitly, via the 'adaptive' character of the RL algorithms of each of the agents. This COIN approach introduces an entirely new, profound design problem: Assuming the RL algorithms are able to achieve high rewards, what reward functions for the individual agents will, when pursued by those agents, result in high world utility? In other words, what reward functions will best ensure that we do not have phenomena like the tragedy of the commons, or Braess's paradox? Although still very young, the science of COINs has already resulted in successes in artificial domains, in particular in packet-routing, the leader-follower problem, and in variants of Arthur's "El Farol bar problem". It is expected that as it matures not only will COIN science expand greatly the range of tasks addressable by human engineers, but it will also provide much insight into already established scientific fields, such as economics, game theory, or population biology.

Wolpert, David H.↗
DOE OSTI2020

Incorporating uncertainty for enhanced leaderboard scoring and ranking in data competitions

Data competitions have become a popular and cost-effective approach for crowdsourcing versatile solutions from diverse expertise. Current practice relies on the simple leaderboard scoring based on a given set of competition data for ranking competitors and distributing the prize. However, a disadvantage of this practice in many competitions is that a slight difference in the scores due to the natural variability of the observed data could result in a much larger difference in the prize amounts. In this article, we propose a new strategy to quantify the uncertainty in the rankings and scores from using different data sets that share common characteristics with the provided competition data. By using a bootstrap approach to generate many comparable data sets, the new method has four advantages over current practice. Furthermore, during the competition, it provides a mechanism for competitors to get feedback about the uncertainty in their relative ranking. After the competition, it allows the host to gain a deeper understanding of the algorithm performance and their robustness across representative data sets. It also offers a transparent mechanism for prize distribution to reward the competitors more fairly with superior and robust performance. Finally, it has the additional advantage of being able to explore what results might have looked like if competition goals evolved from their original choices. The implementation of the strategy is illustrated with a real data competition hosted by Topcoder on urban radiation search.

42 ENGINEERING↗
DOE OSTI2024

Design, Selection, and Evaluation of Reinforcement Learning Single Agents for Ground Target Tracking

Previous approaches for small fixed-wing unmanned air systems that carry strapdown rather than gimbaled cameras achieved satisfactory ground target tracking performance using both standard and deep reinforcement learning algorithms. However, these approaches have significant restrictions and abstractions to the dynamics of the vehicle, such as constant airspeed and constant altitude, because the number of states and actions was necessarily limited. Thus, extensive tuning was required to obtain good tracking performance. The expansion from 4 state–action degrees of freedom to 15 enabled the agent to exploit previous reward functions that produced novel yet undesirable emergent behavior. This paper investigates the causes of and various potential solutions to undesirable emergent behavior in the ground target tracking problem. A combination of changes to the environment, reward structure, action space simplification, command rate, and controller implementation provides insight into obtaining stable tracking results. Consideration is given to reward structure selection and refinement to mitigate undesirable emergent behavior. Results presented in the paper for a simulated environment of a single unmanned air system tracking a randomly moving single ground target show that a soft actor–critic algorithm can produce feasible tracking trajectories without limiting the state space and action space, provided that the environment is properly posed.

Engineering↗
DOE OSTI2026

Financial-technical co-design for capital-intensive, resource-responsive energy systems

Because of their capital-intensive operation, wind energy systems that are competitive in terms of the cost of the energy that they produce lead to risk-reward trade-offs that make their business cases less favorable than those of conventional energy generation technologies. However, wind energy systems tend to be designed to maximize energy production or minimize cost of energy rather than to maximize their business cases. In this work, we attempt to exploit designs specifically tailored to business cases. We develop a novel framework for analyzing energy systems that ties their design variables to monthly operating incomes using simple models and historical hourly market and resource data. Using this approach, we demonstrate that for a wind site with abundant wind resource in the California Independent System Operator market, we can control the trade-off between mean and 5th percentile monthly returns by choosing the specific power of the turbine at a fixed modeled initial capital cost. Our framework gives a measure of the risk-reward spectrum of energy generation assets that could be built at a given site with respect to the sub-annual resource/market variation.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2021

DOE Energy Frontier Research Centers Center for Direct Catalytic Conversion of Biomass to Biofuels (C3Bio)

New capabilities to predict, design and control the chemistries of carbon could answer a global imperative to transition from fossil-based to sustainable transportation fuels. While the use of inexpensive hydrocarbons has been an unparalleled achievement and enabler of economic prosperity for many nations, singular dependence upon crude oil has given rise to systemic vulnerabilities in climate, energy, economic, and national security. Lignocellulosic biomass, a renewable and carbon-neutral resource, has the potential to displace an estimated annual equivalent of three billion barrels of oil in the U.S. alone (National Research Council 2009, U. S. Department of Energy, 2011). However, biomass has only one-third the energy density of crude oil (Agrawal and Singh 2009, Richard 2010) and lacks petroleum’s versatility as a feedstock for fuels and chemicals. These limitations keep biomass conversion below the efficiency level needed for strategic impact while the scientific challenge of routing carbon from one molecular context to another remains unmet. In 2009, the Center for Direct Catalytic Conversion of Biomass to Biofuels (C3Bio) recognized the potential of chemical catalysis and fast pyrolysis to overcome such limitations by transforming the main components of biomass (cellulose, xylan, and lignin) from grasses and trees directly to liquid hydrocarbons and aromatic co-products. Enabled by the EFRC high-risk, high-reward approach to grand challenge science, C3Bio researchers have been key national players in disrupting the conventional paradigm of the cellulosic biorefinery into a new future of “no carbon left behind”—the full utilization of carbon from plant cell walls in energy-dense fuels (Fig. 1). We identified catalytic and fast-pyrolytic pathways that utilize cellulose, xylan and, most significantly, lignin. We developed catalytic processes that deoxygenate and transform monomers and isolated polymers into useful products and tested their use with intact biomass. We gained control of lignin synthesis within plants and initiated tailoring biomass to its end-use through the tools of plant molecular biology and genetic engineering. C3Bio breakthroughs have increased the energy density of biomass-derived substrates via catalytic and pyrolytic conversions into products such as benzoquinones, furfural and hydroxymethylfurfural, levoglucosan and levulinic acid, methoxypropylphenols and propylbenzene. Such advances in biomass conversion would not have been possible without simultaneous advances in analytical instrumentation and methodologies, and imaging technologies and applications. The legacy science developed by C3Bio enables design and control strategies for achieving a targeted product portfolio of fuel and chemical feedstocks from a diverse range of native and tailored biomass. Our research provides the knowledge base required for a bio-economy with product streams as diverse in functionality as those of the petrochemical industry. Coupling targeted computational modeling with experimentation, we achieved: (1) fundamental understanding of biopolymers and cell wall architecture assembly, (2) discovery of new chemistries that allow the development of highly selective pathways to fuels and desirable chemicals, and (3) an integrated systems-level understanding to control catalytic and pyrolytic pathways.

09 BIOMASS FUELS↗
DOE OSTI2022

Variational multiscale reinforcement learning for discovering reduced order closure models of nonlinear spatiotemporal transport systems

Abstract A central challenge in the computational modeling and simulation of a multitude of science applications is to achieve robust and accurate closures for their coarse-grained representations due to underlying highly nonlinear multiscale interactions. These closure models are common in many nonlinear spatiotemporal systems to account for losses due to reduced order representations, including many transport phenomena in fluids. Previous data-driven closure modeling efforts have mostly focused on supervised learning approaches using high fidelity simulation data. On the other hand, reinforcement learning (RL) is a powerful yet relatively uncharted method in spatiotemporally extended systems. In this study, we put forth a modular dynamic closure modeling and discovery framework to stabilize the Galerkin projection based reduced order models that may arise in many nonlinear spatiotemporal dynamical systems with quadratic nonlinearity. However, a key element in creating a robust RL agent is to introduce a feasible reward function, which can be constituted of any difference metrics between the RL model and high fidelity simulation data. First, we introduce a multi-modal RL to discover mode-dependant closure policies that utilize the high fidelity data in rewarding our RL agent. We then formulate a variational multiscale RL (VMRL) approach to discover closure models without requiring access to the high fidelity data in designing the reward function. Specifically, our chief innovation is to leverage variational multiscale formalism to quantify the difference between modal interactions in Galerkin systems. Our results in simulating the viscous Burgers equation indicate that the proposed VMRL method leads to robust and accurate closure parameterizations, and it may potentially be used to discover scale-aware closure models for complex dynamical systems.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1997

Inherit Space

The objective of the proposed research was to begin development of a unique educational tool targeted at educating and inspiring young people 12-16 years old about NASA and the Space Program. Since these young people are the future engineers, scientists and space pioneers, the nurturing of their enthusiasm and interest is of critical importance to the Nation. This summer the basic infrastructure of the tool was developed in the context of an educational game paradigm. The game paradigm has achieved remarkable success in maintaining the interest of young people in a self-paced, student-directed learning environment. This type of environment encourages student exploration and curiosity which are exactly the traits that future space pioneers need to develop to prepare for the unexpected. The Inherit Space Educational Tool is an open-ended learning environment consisting of a finite-state machine classic adventure game paradigm. As the young person explores this world, different obstacles must be overcome. Rewards will be offered such as using the flight simulator to fly around and explore Titan. This simulator was modeled on conventional Earth flight simulators but has been considerably enhanced to add texture mapping of Titan's atmosphere utilizing the latest information from the NASA Galileo Space Probe. Additional scenery was added to provide color VGA graphics of a futuristic research station on Titan as well as an interesting story to keep the youngster's attention. This summer the game infrastructure has been developed as well as the Titan Flight Simulator. A number of other enhancements are planned.

Giarratano, Joseph C.↗
DOE OSTI2022

Giant g-factors and fully spin-polarized states in metamorphic short-period InAsSb/InSb superlattices

Abstract Realizing a large Landé g -factor of electrons in solid-state materials has long been thought of as a rewarding task as it can trigger abundant immediate applications in spintronics and quantum computing. Here, by using metamorphic InAsSb/InSb superlattices (SLs), we demonstrate an unprecedented high value of g ≈ 104, twice larger than that in bulk InSb, and fully spin-polarized states at low magnetic fields. In addition, we show that the g -factor can be tuned on demand from 20 to 110 via varying the SL period. The key ingredients of such a wide tunability are the wavefunction mixing and overlap between the electron and hole states, which have drawn little attention in prior studies. Our work not only establishes metamorphic InAsSb/InSb as a promising and competitive material platform for future quantum devices but also provides a new route toward g -factor engineering in semiconductor structures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Game-Theoretic Strategies for Cyber-Physical Infrastructures Under Component Disruptions

In this work, networked infrastructures of recursively defined systems composed of discrete cyber and physical components are considered. The components of basic systems at the finest levels can be disrupted by cyber or physical means, and can be reinforced to survive at certain costs. A problem of ensuring the infrastructure performance is formulated as a game between a provider and an attacker, who probabilistically choose components to reinforce and attack, respectively. The disruptions of this infrastructure are characterized using the aggregate failure correlation function that specifies the conditional failure probability of the infrastructure given the failure of an individual system at that level. The survival probabilities of basic systems satisfy simple product-form, first-order differential equations expressed in terms of the multiplier functions. The utility functions of the provider and attacker are composed of the reward and cost terms, both expressed in terms of the component reinforcement and attack probabilities. The Nash equilibrium of this game is characterized, along with the sensitivity functions of the survival probabilities of basic systems that highlight their dependence individually on the cost-benefit terms, the correlation functions, and the multiplier functions. These results are illustrated using simplified models of a distributed cloud servers infrastructure, a 5G data network infrastructure, a high performance computing federation, and a smart energy grid infrastructure.

42 ENGINEERING↗
DOE OSTI2022

Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner

We report in this paper describes how the performance of motion primitive-based planning algorithms can be improved using reinforcement learning. Specifically, we describe and evaluate a framework that autonomously improves the performance of a primitive-based motion planner. The improvement process consists of three phases: exploration, extraction, and reward updates. This process can be iterated continuously to provide successive improvement. The exploration step generates new trajectories, and the extraction step identifies new primitives from these trajectories. These primitives are then used to update rewards for continued exploration. This framework required novel shaping rewards, development of a primitive extraction algorithm, and modification of the Hybrid A* algorithm. The framework is tested on a navigation task using a nonlinear F-16 model. The framework autonomously added 91 motion primitives to the primitive library and reduced average path cost by 21.6 seconds, or 35.75% of the original cost. The learned primitives are applied to an obstacle field navigation task, which was not used in training, and reduced path cost by 16.3 seconds, or 24.1%. Additionally, two heuristics for the modified Hybrid A* algorithm are designed to improve effective branching factor.

42 ENGINEERING↗