Search NASA⌕ Search

SEARCH · Search NASA

Results for “reward engineering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Deep reinforcement learning for dynamic control of fuel injection timing in multi-pulse compression ignition engines

Conventional compression-ignition (CI) engines have long offered high thermal efficiencies and torque across a wide range of loads, but often require extensive exhaust gas treatment that decreases efficiency to meet ever-increasing emissions regulations. One strategy to decrease emissions is to split the fuel injection into a series of smaller injections. In this paper, we explore a new way of discovering optimal control strategies for the next generation of CI engines using deep reinforcement learning (DRL). We outline a DRL procedure to maximize the weighted reward of engine work while minimizing end-of-cycle NO x emissions. Through the procedure outlined in this paper, we show that the DRL agent is able to reduce NO x emissions threefold while only decreasing network by 2%. We demonstrate the use of transfer learning (TL) across hierarchies of physical models to accelerate the learning process, making this approach feasible for a range of control problems within this space. This paper presents a framework and demonstration for using DRL to design control systems in technology areas such as multi-pulse engine control where a hierarchy of models combined with multi-objective rewards are used for optimal operation.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2021

Exploring electron beam induced atomic assembly via reinforcement learning in a molecular dynamics environment

We report atom-by-atom assembly of functional materials and devices is perceived as one of the ultimate targets of nanotechnology. Recently it has been shown that the beam of a scanning transmission electron microscope can be used for targeted manipulation of individual atoms. However, the process is highly dynamic in nature rendering control difficult. One possible solution is to instead train artificial agents to perform the atomic manipulation in an automated manner without need for human intervention. As a first step to realizing this goal, we explore how artificial agents can be trained for atomic manipulation in a simplified molecular dynamics environment of graphene with Si dopants, using reinforcement learning. We find that it is possible to engineer the reward function of the agent in such a way as to encourage formation of local clusters of dopants under different constraints. This study shows the potential for reinforcement learning in nanoscale fabrication, and crucially, that the dynamics learned by agents encode specific elements of important physics that can be learned.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2024

Integrating an Ensemble Reward System into an Off-Policy Reinforcement Learning Algorithm for the Economic Dispatch of Small Modular Reactor-Based Energy Systems

Nuclear Integrated Energy Systems (NIES) have emerged as a comprehensive solution for navigating the changing energy landscape. They combine nuclear power plants with renewable energy sources, storage systems, and smart grid technologies to optimize energy production, distribution, and consumption across sectors, improving efficiency, reliability, and sustainability while addressing challenges associated with variability. The integration of Small Modular Reactors (SMRs) in NIES offers significant benefits over traditional nuclear facilities, although transferring involves overcoming legal and operational barriers, particularly in economic dispatch. This study proposes a novel off-policy Reinforcement Learning (RL) approach with an ensemble reward system to optimize economic dispatch for nuclear-powered generation companies equipped with an SMR, demonstrating superior accuracy and efficiency when compared to conventional methods and emphasizing RL’s potential to improve NIES profitability and sustainability. Finally, the research attempts to demonstrate the viability of implementing the proposed integrated RL approach in spot energy markets to maximize profits for nuclear-driven generation companies, establishing NIES’ profitability over competitors that rely on fossil fuel-based generation units to meet baseload requirements.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

Physics-based reward driven image analysis in microscopy

The rise of electron microscopy has expanded our ability to acquire nanometer and atomically resolved images of complex materials. The resulting vast datasets are typically analyzed by human operators, an intrinsically challenging process due to the multiple possible analysis steps and the corresponding need to build and optimize complex analysis workflows. We present a methodology based on the concept of a Reward Function coupled with Bayesian Optimization, to optimize image analysis workflows dynamically. The Reward Function is engineered to closely align with the experimental objectives and broader context and is quantifiable upon completion of the analysis. Here, cross-section, high-angle annular dark field (HAADF) images of ion-irradiated (Y, Dy)Ba 2 Cu 3 O 7–δ thin-films were used as a model system. The reward functions were formed based on the expected materials density and atomic spacings and used to drive multi-objective optimization of the classical Laplacian-of-Gaussian (LoG) method. These results can be benchmarked against the DCNN segmentation. This optimized LoG* compares favorably against DCNN in the presence of the additional noise. We further extend the reward function approach towards the identification of partially-disordered regions, creating a physics-driven reward function and action space of high-dimensional clustering. We pose that with correct definition, the reward function approach allows real-time optimization of complex analysis workflows at much higher speeds and lower computational costs than classical DCNN-based inference, ensuring the attainment of results that are both precise and aligned with the human-defined objectives.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states

Maintenance policy optimization usually is faced with challenges that arise from an imperfect knowledge of system degradation models and from the partial observability of system degradation states. Here, this paper proposes a reinforcement learning method to address these two challenges for a class of maintenance problems with Markov degradation processes. The reinforcement learning approach consists of a learning component and a planning component. Using sequentially collected observations, at each step of decision-making the learning component improves the knowledge of system degradation in terms of the probability distributions of the transition rates based on sequential Bayesian inference. Using the updated transition rates, at each step of decision-making the maintenance policy optimization problem is then formulated as a partially observable Markov decision problem, and the planning component computes the optimal maintenance policy that maximizes the expected cumulative reward. The proposed method is illustrated using a numerical example with repair and inspection maintenance actions. The result shows that as more observations are collected, the learning component progressively learns the true system degradation process, and the planning component adjusts the optimal maintenance policy accordingly as well, which leads to increased reward.

42 ENGINEERING↗
DOE OSTI2023

Matching diverse feedstocks to conversion processes for the future bioeconomy

A wide variety of wasted or underutilized organic feedstocks can be leveraged to build a sustainable bioeconomy, ranging from crop residues to food processor residues and municipal wastes. Leveraging these feedstocks is both high-risk and high-reward. Converting mixed, variable, and/or highly contaminated feedstocks can pose engineering and economic challenges. However, converting these materials to fuels and chemicals can divert waste from landfills, reduce fugitive methane emissions, and enable more responsible forest management to reduce the frequency and severity of wildfires. Historically, low-value components, including ash and lignin, are poised to become valuable coproducts capable of supplementing cement and valuable chemicals. Here, we evaluate the challenges and opportunities associated with converting a range of feedstocks to renewable fuels and chemicals.

09 BIOMASS FUELS↗
DOE OSTI2020

Batteries Annual Progress Report (FY2019)

The Vehicle Technologies Office (VTO) of the Department of Energy (DOE) conducts research and development (R&D) on advanced transportation technologies that would reduce the nation’s use of imported oil and would also lead to reductions in harmful emissions. Technologies supported by VTO include electric drive components such as advanced energy storage devices (primarily batteries), power electronics and electric drive motors, advanced structural materials, energy efficient mobility systems, advanced combustion engines, and fuels. VTO is focused on funding early-stage high-reward/high-risk research to improve critical components needed for more fuel efficient (and cleaner-operating) vehicles. One of the major VTO objectives is to enable U.S. innovators to rapidly develop the next generation of technologies that achieve the cost, range, and charging infrastructure necessary for the widespread adoption of plug-in electric vehicles (PEVs). An important prerequisite for the electrification of the nation’s light duty transportation sector is development of more cost-effective, longer lasting, and more abuse-tolerant PEV batteries. One of the ultimate goals of this research, consistent with the current vehicle electrification trend, is an EV which can provide the full driving performance, convenience, and price of an internal combustion engine (ICE) vehicle. To achieve this, VTO has established the following overarching goal (Source: FY2021 Congressional Budget Justification1): …identify new battery chemistry and cell technologies with the potential to reduce the cost of electric vehicle battery packs by more than half, to less than $100/kWh (ultimate goal is $60/kWh battery cell cost), increase range to 300 miles, and decrease charge time to 15 minutes or less by 2028. VTO works with key U.S. automakers through the United States Council for Automotive Research (USCAR) – an umbrella organization for collaborative research consisting of Fiat Chrysler Automobiles (FCA), the Ford Motor Company, and General Motors. Collaboration with automakers through the partnership known as U.S. Driving Research and Innovation for Vehicle Efficiency and Energy Sustainability (U.S. DRIVE) attempts to enhance the relevance and the success potential of its research portfolio. VTO competitively selects projects for funding through funding opportunity announcements (FOAs). Directly-funded work at the national laboratories are awarded competitively through a lab-call process. During the past year, VTO continued R&D in support of PEVs. Stakeholders for VTO R&D include universities, national laboratories, other government agencies and industry (including automakers, battery manufacturers, material suppliers, component developers, private research firms, and small businesses). This document summarizes the progress of VTO battery R&D projects supported during the fiscal year 2019 (FY 2019).

25 ENERGY STORAGE↗
DOE OSTI2025

Entanglement engineering of optomechanical systems by reinforcement learning

Entanglement is fundamental to quantum information science and technology, yet controlling and manipulating entanglement—so-called entanglement engineering—for arbitrary quantum systems remains a formidable challenge. There are two difficulties: the fragility of quantum entanglement and its experimental characterization. We develop a model-free deep reinforcement-learning (RL) approach to entanglement engineering, in which feedback control together with weak continuous measurement and partial state observation is exploited to generate and maintain desired entanglement. We employ quantum optomechanical systems with linear or nonlinear photon–phonon interactions to demonstrate the workings of our machine-learning-based entanglement engineering protocol. In particular, the RL agent sequentially interacts with one or multiple parallel quantum optomechanical environments, collects trajectories, and updates the policy to maximize the accumulated reward to create and stabilize quantum entanglement over an arbitrary amount of time. The machine-learning-based model-free control principle is applicable to the entanglement engineering of experimental quantum systems in general.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

The Necessity of International Particle Physics Opportunities for American Education

This Snowmass2021 Contributed Paper addresses the role of the Particle Physics community in creating and fostering international connections in American education. It describes the pressing need to introduce students and faculty to the challenges and rewards of international collaboration, not only to develop the next generation of scientists and engineers for particle physics, but to maintain and build U.S. leadership on an increasingly competitive world stage. We present and assess current efforts in education and public engagement with an eye toward identifying those activities in need of change or increased resources to improve audience reach and program efficacy. We also consider possible new activities that might improve upon or complement existing programs, with the common goal of providing all U.S. students with the opportunity to benefit from a quality international scientific experience.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Center for direct catalytic conversion of biomass to biofuels (Final Technical Report)

Lignocellulosic biomass has only one-third the energy density of crude oil and lacks petroleum’s versatility as a feedstock for fuels and chemicals. Since 2009, the Center for direct Catalytic Conversion of Biomass to Biofuels (C3Bio) has recognized the potential of chemical catalysis and fast-pyrolysis to overcome such limitations by transforming the main components of biomass (cellulose, xylan, and lignin) from grasses and trees directly to liquid hydrocarbons and aromatic co-products. In 2014, C3Bio proposed to develop critical systems-level understanding of how biomass structural complexity at molecular, nanoscale, and mesoscale levels impacts the yields and selectivities of desired products from catalytic and pyrolytic transformations. Our long-term goal was to gain unprecedented control of effective routing of carbon: we aimed to specify both the structures within, and the reaction products from, lignocellulosic biomass. Enabled by the EFRC high-risk, high-reward approach to grand challenge science, our interdisciplinary team of plant biologists, chemists, and chemical engineers disrupted the conventional paradigm of the cellulosic biorefinery into a new future of “no carbon left behind” - the full utilization of carbon from plant cell walls in energy-dense fuels.

09 BIOMASS FUELS↗
DOE OSTI2024

Spiking Markov Reward Process v.0.1

SAND2024-11150O The Spiking Markov Reward Process software is a spiking neural network that streams binary arithmetic and computes the state value function of a Markov reward process. The software will be released to the SpiNNcloud group for development of neuromorphic acceleration. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Wang, Felix↗
DOE OSTI2022

For the Public Good: Connecting, Retaining, and Recognizing Current and Future RSEs at U.S. National Research Laboratories and Agencies

U.S. national research laboratories and agencies play an integral role in advancing science and technology for the public good. The authors of this article, as research software engineers (RSEs) and allies from eight unique national R&D organizations, came together to explore RSE needs from the perspective of national institutions. We identified three key areas of improvement for future RSEs to pursue science in the national interest: community establishment, hiring and retention, and recognition. In conclusion, to retain and cultivate this essential talent, U.S. national institutions must evolve to support appropriate career pathways for RSEs, and to recognize and reward RSEs’ work.

42 ENGINEERING↗
DOE OSTI2024

Los Alamos National Laboratory Laboratory Directed Research and Development Program (FY22 Annual Progress Report)

Laboratory Directed Research and Development (LDRD) helps Los Alamos National Laboratory solve national security challenges through excellence in mission-focused science, technology, and engineering. The DOE Laboratory and Site-Directed Research and Development programs are among the most impactful sources of research and development for our Nation. The heart of the LDRD program is high-risk, high-reward research that creates innovative technical solutions for some of the Nation’s most difficult challenges. The LDRD programs follow strategic guidance derived from the missions of the U.S. Department of Energy, the National Nuclear Security Administration (NNSA), and the Laboratory.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2026

Los Alamos National Laboratory Laboratory Directed Research and Development Program: FY25 Annual Progress Report

Laboratory Directed Research and Development (LDRD) helps Los Alamos National Laboratory solve national security challenges through excellence in mission-focused science, technology, and engineering (ST&E). The Department of Energy (DOE) Laboratory and Site-Directed Research and Development programs are among the most impactful sources of research and development for our Nation. The heart of the LDRD program is high-risk, high-reward research that creates innovative technical solutions for some of the Nation’s most difficult challenges. The LDRD programs follow strategic guidance derived from the missions of the U.S. Department of Energy, the National Nuclear Security Administration (NNSA), and the Laboratory.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2023

Nuclear-Renewable-Storage Systems: Enhancing Planning and Operations of Integrated Energy Systems

Nuclear-renewable-storage integrated energy systems (IES) are multi-carrier energy systems that include not only electricity but also other forms of demands. Because individual IES components must observe their thermo-physical limits, including ramp rates, start-up, and shut-down time, we formulate operations of the IES as an optimization model by minimizing the total operations costs subject to physical limits of all constituent components. In addition, we develop a data-driven approach to improve the computational performance of the economic dispatch model by using reinforcement learning, where an agent is rewarded by meeting demands and penalized otherwise when shifting to the next state.

25 ENERGY STORAGE↗
DOE OSTI2022

aphBO-2GP-3B: a budgeted asynchronous parallel multi-acquisition functions for constrained Bayesian optimization on high-performing computing architecture

High-fidelity complex engineering simulations are often predictive, but also computationally expensive and often require substantial computational efforts. The mitigation of computational burden is usually enabled through parallelism in high-performance cluster (HPC) architecture. Optimization problems associated with these applications is a challenging problem due to the high computational cost of the high-fidelity simulations. In this paper, an asynchronous parallel constrained Bayesian optimization method is proposed to efficiently solve the computationally expensive simulation-based optimization problems on the HPC platform, with a budgeted computational resource, where the maximum number of simulations is a constant. The advantage of this method are three-fold. Firstly, the efficiency of the Bayesian optimization is improved, where multiple input locations are evaluated parallel in an asynchronous manner to accelerate the optimization convergence with respect to physical runtime. This efficiency feature is further improved so that when each of the inputs is finished, another input is queried without waiting for the whole batch to complete. Second, the proposed method can handle both known and unknown constraints. Third, the proposed method samples several acquisition functions based on their rewards using a modified GP-Hedge scheme. The proposed framework is termed aphBO-2GP-3B, which means asynchronous parallel hedge Bayesian optimization with two Gaussian processes and three batches. The numerical performance of the proposed framework aphBO-2GP-3B is comprehensively benchmarked using 16 numerical examples, compared against other 6 parallel Bayesian optimization variants and 1 parallel Monte Carlo as a baseline, and demonstrated using two real-world high-fidelity expensive industrial applications. The first engineering application is based on finite element analysis (FEA) and the second one is based on computational fluid dynamics (CFD) simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

IMPROVING TRANSPORTATION EFFICIENCY THROUGH INTEGRATED VEHICLE, ENGINE, AND POWERTRAIN RESEARCH -SUPERTRUCK II

Daimler Truck North America (DTNA) completed a five year, $40.1 M SuperTruck 2 project to demonstrate technologies to achieve both vehicle and engine efficiency improvements. The vehicle objective was to develop and demonstrate a concept vehicle with at least 115% vehicle freight efficiency improvement over a weighted average of four cycles relative to a 2009 best-in-class baseline vehicle. The engine technology development goal was to achieve 55% brake thermal efficiency (BTE) as tested on a dynamometer at an equivalent of 65 MPH. Both project objectives were to develop technologies that are cost effective. SuperTruck 2 started in 2017 and built on the knowledge gained in SuperTruck 1. SuperTruck 2 enabled the ability to evaluate high risk high reward research centered on four areas that show the most potential for commercialization: aerodynamics, powertrain, rolling resistance and energy management. In addition, several industry, University and National laboratories were able to collaborate in developing technologies found to be successful in meeting the program objectives. Several technologies show promise towards production while some technologies do not show a quick path towards production.

Villeneuve, Darek↗
DOE OSTI2020

Optimal carbon storage reservoir management through deep reinforcement learning

Model-based optimization plays a central role in energy system design and management. The complexity and high-dimensionality of many process-level models, especially those used for geosystem energy exploration and utilization, often lead to formidable computational costs when the dimension of decision space is also large. This work adopts elements of recently advanced deep learning techniques to solve a sequential decision-making problem in applied geosystem management. Specifically, a deep reinforcement learning framework was formed for optimal multiperiod planning, in which a deep Q-learning network (DQN) agent was trained to maximize rewards by learning from high-dimensional inputs and from exploitation of its past experiences. To expedite computation, deep multitask learning was used to approximate high-dimensional, multistate transition functions. Both DQN and deep multitask learning are pattern based. As a demonstration, the framework was applied to optimal carbon sequestration reservoir planning using two different types of management strategies: monitoring only and brine extraction. Both strategies are designed to mitigate potential risks due to pressure buildup. Results show that the DQN agent can identify the optimal policies to maximize the reward for given risk and cost constraints. Finally, experiments also show that knowledge the agent gained from interacting with one environment is largely preserved when deploying the same agent in other similar environments.

15 GEOTHERMAL ENERGY↗