Search NASASearch

SEARCH · Search NASA

Results for “policy optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Virtual to Physical: Reinforcement Learning to Optimize SNS Particle Accelerator Controls

Complex accelerators must have control systems that can handle dynamic nonlinear environments. This makes traditional control methods unsuitable as they can struggle to adapt to these uncertainties. This provides an ideal environment for reinforcement learning algorithms as they are adaptable and generalizable. We present a reinforcement learning pipeline that can effectively handle the dynamics of a complex accelerator. We test and prove our pipelines capabilities on multiple environments including the Spallation Neutron Source (SNS) and the Beam Test Facility (BTF) at Oakridge National Lab (ORNL). Due to the limited time available to train an online algorithm like reinforcement learning on a real accelerator, we utilize a virtual twin accelerator (VIRAC) developed by ORNL to pretrain the policy and show its ability to converge in the virtual environment. We then test the adaptability of the pretrained RL model by applying it on the real accelerator and comparing the results. Utilizing our Scientific Optimization and Controls Toolkit (SOCT) and open-source standards such as Gymnasium we create and solve for a MEBT orbit correction problem in the SNS and an emittance maximization problem in the BTF. We show how Twin Delayed Deep Deterministic Policy Gradient (TD3) can solve this optimization environment in the virtual accelerator and transfer this policy onto the real accelerator for inference and model retraining. We show how reinforcement learning can be utilized as a control system for complex accelerators and provide a model pipeline for how an implementation performs and can be adapted to new accelerator control problems.

Kasparian, Armen [Thomas Jefferson National Accele

Scientific frontiers of agrivoltaic cropping systems

Agrivoltaic (AV) systems integrate agriculture with electricity conversion through photovoltaic (PV) modules. Compared with conventional ground-mounted PV systems, AV systems can reduce land-use competition and offer agronomic and economic advantages, such as more stable crop production and additional farm income. However, AV systems can decrease agricultural performance and are typically 20-90% costlier to install than conventional PV systems. Here, in this Review, we analyse the implementation of AV cropping systems to preserve agricultural activities and highlight challenges and barriers. The global electricity potential of AV systems is ~66-385 PWh annually, depending on PV technology and installation density, if deployed in the most suitable areas, without accounting for grid availability. Scaling up has been hindered by crop selection for shading conditions, decreased energy conversion per unit of land area and issues with social acceptance, landscape impact and environmental sustainability. These issues can be addressed by developments such as wavelength-selective PV; system configurations, such as optimizing module spacing to reduce shading; and operational methods, such as optimizing tracking strategies and integrating agricultural infrastructure. Cross-sector policies can support AV systems by addressing the needs of diverse stakeholders over shared land resources. Further development will require collaboration among the design, performance, deployment and systems research communities.

14 SOLAR ENERGY

Data and Code for 'GHG Mitigation and Land Use Change Implications of Sustainable Aviation Fuel in the United States'

BEPAM, Biofuel and Environmental Policy Analysis Model, models the agricultural sector and determines economically optimal land-use and feedstock mix at the US scale by maximizing the sum of agricultural sector consumers’ and producers’ surplus subject to various resource balances, land availability, and technological constraints under a range of biomass prices, from zero to $140 Mg-1 over the 2016-2030 period. Here BEPAM is used to model SAF production using energy crops and crop residues. BEPAM uses the GAMS format and uses yield and GHG balance projections from the biogeochemical model, DayCent.

09 BIOMASS FUELS

Component-to-Optimization Workflow Demonstration

This report aims to demonstrate workflow-generating algorithms for optimizing dispatch across a broad range of Integrated Energy System applications using the Framework for Optimization of Resources and Economics (FORCE) tool suite. The optimization is performed at two different time scales. In the coarse time scale, the optimization focuses on a class of energy sources and consumers and aims to find the optimal combinations and flows of energy based on real-time price data information. In the fine time scale, the optimization focuses on a specific thermal energy delivery system and aims to find the optimal setpoints of components in order to meet the energy demands from coarse-time-scale optimizations. In this demonstration, the coarse-scale optimization is implemented using the newly developed Dispatch Optimization Variable Engine (DOVE), while the fine-scale optimization used Optimization of Real-Time Capacity Allocation (ORCA).

29 ENERGY PLANNING, POLICY, AND ECONOMY

Hydropower Flexibility Framework (Final Technical Report)

The Hydropower Flexibility Framework (HFF) tool focuses on providing the hydropower community with an effective means of assessing optimized hydropower plant outcomes. This tool combines both site specific characteristics, which act to constrain plant operation, and the hydrologic and grid characteristics which drive hydropower plant operation. The hydropower community faces a confluence of factors which drive the importance of developing such a capability, including an aging hydropower fleet subject to a range of modernization opportunities, a large number of hydropower plant relicensing activities which may affect operational requirements, an electrical grid with increasing levels of variable resources which must be balanced to maintain grid stability, and climate change influencing riverine hydrologic patterns outside of design characteristics. With support from the hydropower community, the project team developed the HFF tool and demonstrated the tool through a series of Use Cases. This guidance was developed as a part of the larger HFF tool User’s Manual (see Appendix B), a resource designed to inform other users and to empower community uptake of the tool. The HFF tool, hosted at https://hfftool.com/, was developed with the support of the U.S. Department of Energy (DOE) Water Power Technologies Office (WPTO). EPRI is currently exploring alternatives to support the continued maintenance and functionally of the online tool.

13 HYDRO ENERGY

Application of Banking Scoring and Rating for Coherent Risk Measures in Electricity Systems ABSCORES

This project developed a framework for asset and system risk management that can be incorporated into current electricity system operations to improve economic efficiency and establish an Electric Assets Risk Bureau. We leveraged scoring and ratings from banking and financial institutions alongside current optimization methods in dispatching power systems to help system operators and electricity markets schedule resources. This approach is based on the observation that there are major discrepancies between the power scheduled by a system operator and the actual power generated/consumed. These discrepancies—exacerbated by unplanned contingencies (e.g., natural disasters)—are caused by multiple factors, including the different financial, environmental and risk preferences of power producers, consumers, and aggregators. We developed a framework that counteracts two failures in electricity system operations: imperfect information and missing markets for products. The technical approach included five tasks. Tasks 1 and 2 supported the development of risk scores at the asset level with historical data collected for this project. Tasks 3, 4, and 5 incorporated scoring into decision-making at the system level. The proposed effort achieved PERFORM's Program Objectives because the proposed outputs and algorithms do not exist in the electricity industry and are an innovative approach to managing risk. Since the acknowledged need to better assess and act upon risk profiles for grid assets has not been met by the industry, this project will also impact ARPA-E's Mission Areas, including improving energy efficiency and giving the U.S. a technological lead in advanced energy technologies.

29 ENERGY PLANNING, POLICY, AND ECONOMY

CTRL-STEER: Closed-Loop Neuron Activation Control in Vision-Language-Action Models

Vision-Language-Action (VLA) models enable test-time behavioral steering via neuron-level interventions, but existing methods use fixed strengths and operate in open loop. This static modulation fails under evolving task dynamics, leading to overcorrection, oscillations, and reduced task success—especially for temporal attributes like speed. We propose CTRL-STEER, a control-theoretic framework that casts activation steering as closed-loop feedback with adaptive, time-varying interventions. Instead of assuming neurons encode temporal concepts, we steer along motion-aligned residual directions and regulate intervention magnitude via feedback. We instantiate this with both PID and reinforcement learning controllers that jointly optimize concept adherence and task success. Experiments on fine-tuned OpenVLA policies across four LIBERO suites show improved stability and a better steering–success trade-off over fixed-coefficient baselines, without retraining the base model.

Babu, Abhijith [Florida International University,

Artificial-intelligence-driven shot reduction in quantum measurement

Variational Quantum Eigensolver (VQE) provides a powerful solution for approximating molecular ground state energies by combining quantum circuits and classical computers. However, estimating probabilistic outcomes on quantum hardware requires repeated measurements (shots), incurring significant costs as accuracy increases. Optimizing shot allocation is thus critical for improving the efficiency of VQE. Current strategies rely heavily on hand-crafted heuristics requiring extensive expert knowledge. This paper proposes a reinforcement learning (RL)-based approach that automatically learns shot assignment policies to minimize total measurement shots while achieving convergence to the minimum of the energy expectation in VQE. The RL agent assigns measurement shots across VQE optimization iterations based on the progress of the optimization. This approach reduces VQE's dependence on static heuristics and human expertise. When the RL-enabled VQE is applied to a small molecule, a shot reduction policy is learned. The policy demonstrates transferability across systems and compatibility with other wavefunction Ansätze. In addition to these specific findings, this work highlights the potential of RL for automatically discovering efficient and scalable quantum optimization strategies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH

Agrivoltaics: Unlocking the Potential of Dual Land Use

This tutorial will delve into the practical and technical considerations for agrivoltaic systems, including crop selection, agricultural practices, and solar energy optimization. Leveraging insights from successful case studies, we'll address challenges such as policy barriers and regulatory gaps while exploring opportunities and incentives for implementation. With a focus on PV expertise, attendees will gain actionable knowledge on designing and evaluating dual-use systems that balance energy generation with agricultural productivity.

14 SOLAR ENERGY

Design and optimization of processes for recovering rare earth elements from end‐of‐life permanent magnets

Recovery of rare earth elements (REEs) from end-of-life (EOL) products represents a strategic opportunity to strengthen the domestic supply chain for rare earth elements. This work presents a superstructure-based optimization framework for finding the most economical processing pathway for different EOL rare earth permanent magnets (REPMs). The framework evaluates state-of-the-art technologies across four processing stages—disassembly, demagnetization, leaching and extraction, and precipitation and calcination—using net present value (NPV) maximization and cost of recovery (COR) minimization objectives. A novel bottom-up costing framework for hydrogen decrepitation is also introduced. Two feedstocks were considered: REPMs from EOL hard disk drives (HDDs), and electric and hybrid electric vehicles (EVs and HEVs). While HDD recycling proved unprofitable due to limited feedstock availability, EVs/HEVs were profitable across a range of parameters and cost estimates. Therefore, our findings suggest that the proposed EOL EV/HEV recycling process may be economical and is worthy of further investigation.

29 ENERGY PLANNING, POLICY, AND ECONOMY

Supporting ARPA-E Power Grid Optimization (Final Report)

Pacific Northwest National Laboratory (PNNL), Arizona State University (ASU), Georgia Institute of Technology (Georgia Tech), Los Alamos National Laboratory (LANL), National Renewable Energy Laboratory (NREL), Texas A&M University (TAMU), The University of Texas at Austin (UT), and the University of Wisconsin-Madison (UW-M) supported the ARPA-E Grid Optimization (GO) Competition by providing a common problem formulation, data format, datasets, evaluation mechanism, scoring, rules, and results that resulted in the awarding of $\$9.24$ million dollars to teams from academia, industry, and national labs for solving three sets of increasingly difficult non-linear, security- constrained AC Optimal Powerflow (AC-OPF) optimization problems in order to increase the efficiency of the US Electric Grid. It is estimated that a 1% increase in efficiency can save $\$1$ billion. Current industry practices typically use a linear DC model (DC-OPF) in order solve the OPF problem within the time constraints of the operation schedule. The GO Competition challenges the best power engineers, mathematicians, and computer scientists to make possible operational decisions based on accurate physical models. To accomplish this, the GO Competition created a series of Challenges and funded teams to produce the best solver. Challenge 1 was to solve the security constrained Alternating Current Optimal Power Flow (ACOPF) problem. Challenge 2 extended that to by adding adjustable transformer tap ratios, phase shifting transformers, switchable shunts, price-responsive demand, ramp rate constrained generators and loads, and fast-start unit commitment (UC). Furthermore, Challenge 2 was a maximization problem while Challenge 1 was a minimization problem. While Challenge 3 was being developed, the entrants were invited to find better solutions to the Challenge 2 synthetic datasets with no restrictions on time, hardware, or algorithms. The Challenge 2 solutions turned out to be very good. Challenge 3 expanded the Challenge 2 problem further by using multiperiod dynamic markets, including advisory models for extreme weather events, day-ahead markets, and the real-time markets with an extended look-ahead. These problems included active bid-in demand and topology optimization. Together the Challenges used nearly 30 million CPU hours. Since each team was working on the same problem, using the same data, and running on the same hardware, fair comparisons could be drawn as to the best solver. The datasets were varied enough, however, that the best solver for one dataset was not necessarily the best at another, so cumulative scores were used. The process was managed by the PNNL maintained website https://GOCompetition.energy.gov, where Entrants could find information about the problem, the data, the rules, submit their solver for evaluation, and see the scores of all the competing teams on a Leaderboard. Interest was world-wide but only American teams were eligible for prizes. The Competition has produced 34 journal articles 115 papers and been cited over 500 times in the literature, including 12 dissertations (4 from foreign countries; Columbia (2), Germany, and Italy) and 3 from the DOE ExaScale project. Software developed by Pearl Street Technologies for Challenges 1 and 2 is now deployed by Southwest Power Pool (SPP) and Midcontinent Independent Service Operator (MISO). Other teams have received inquiries from venture capitalists. Google DeepMind has thanked the Competition for making the datasets developed for the Competition public. They are using it to train machine learning models. The larger datasets have billions of unknowns to be solved for, but only a small percent matter in the final solution. Knowing what unknowns are important can dramatically speedup the solution.

24 POWER TRANSMISSION AND DISTRIBUTION

To What Extent Will Decarbonization Deepen the Conversation Between Industry and the Grid?

Decarbonization - the transition away from un-mitigated fossil fuel combustion throughout the economy - requires big changes from both power and process systems. On the power system side, those changes are expected to include large increases in variable generation, e.g., from wind and solar, which has near-zero marginal costs and at large shares can produce infrequent but consequential energy droughts. On the process systems side, industries are investigating their options for direct and indirect electrification, the latter exemplified by replacing fossil fuel inputs with zero-carbon, energy-carrying chemicals like hydrogen and ammonia produced via electrochemical processes. The economic features of these changes within the larger context of power and process systems suggest that their realization could be accompanied by a paradigm shift in how industrial facilities interact with the grid. For example, the dominant type of demand participation in power markets could change from today's focus on load reductions at peak times to a new focus on shifting electricity use, enabled in part by large-scale product storage, to take advantage of renewable energy that would otherwise be curtailed and to avoid consumption during high-price energy droughts. This talk will describe these and other possible design and operational approaches from grid and industrial economic perspectives, culminating in an enumeration of open problems that lie at the interface of today and tomorrow's power and process systems.

co-design

Assimilating partial observation to enhance feedback control of stochastic dynamical systems

Here, in this paper, we present a novel methodology to tackle feedback optimal control problems in scenarios where the exact state of the controlled process is unknown. It integrates data assimilation techniques and optimal control solvers to manage partial observation of the state process, a common occurrence in practical scenarios. Traditional stochastic optimal control methods assume full state observation, which is often not feasible in real-world fluid dynamics control problems. Our approach underscores the significance of utilizing observational data to inform control policy design. Specifically, we introduce a kernel learning backward stochastic differential equation (SDE) filter to enhance data assimilation efficiency and propose a sample-wise stochastic optimization method within the stochastic maximum principle framework. We demonstrate the efficacy and accuracy of our method in the control of advection-diffusion-reaction flow problem and the Dubins airplane maneuvering problem with model uncertainty.

data driven

What Can We Learn from US National Transmission Studies?

Multiple US regions have announced new electricity transmission plans, are actively revisiting the future role of transmission, and could benefit from findings from recent national-scale transmission studies. A review of seven national studies shows that significant transmission expansion is cost-optimal for serving new demand, integrating additional resources, and supporting grid reliability. A majority of the reviewed scenarios, which span a range of policy and technology assumptions, include at least a doubling of the transmission system by 2050. The national studies find substantial net system cost savings from transmission expansion and high benefit-cost ratios for transmission investments. The results are derived through multi-value, nationally coordinated, co-optimized, multi-scenario modeling, which is an approach that could yield lower-cost pathways compared to current industry planning practices.

24 POWER TRANSMISSION AND DISTRIBUTION

Optimizing district energy systems under uncertainty: Insights from a case study from Washington D.C., USA

This study investigates solutions for delivering affordable heating and cooling to a brownfield site, focusing on a case study in Washington, DC. Moving towards more diverse and resilient energy systems, we identify the optimal portfolio for a district energy system with diverse energy sources to meet the area’s energy demands. Our methodological approach integrates two detailed models: one calculating building-level energy demand and the other optimizing district energy technology choices based on their demand profiles, accounting for uncertainties in energy prices, policies, and other parameters. The results provide an economic comparison of district and individual supply options at the building level, emphasizing the flexibility district systems can offer to the electricity sector. District energy systems demonstrate cost-stabilization benefits amidst volatile energy prices and external uncertainties. For heating, district systems yield significant cost savings compared to individual solutions, driven by fuel flexibility and the use of local renewable energy sources. For cooling, district systems also show advantages, though individual systems may remain more cost-effective for smaller buildings. Additionally, district systems exhibit considerable flexibility on the heating side, as evidenced by variations in electricity consumption. We recommend future research to explore the relationship between the economics of district energy systems, particularly at the building level, and their flexibility potential for the electricity sector across diverse geographic contexts to reduce overall grid costs and promote grid reliability. This includes areas with distinct zoning laws, municipal priorities, utility structures, and funding mechanisms, such as the United States, and regions like Europe with pronounced electricity price volatility.

24 POWER TRANSMISSION AND DISTRIBUTION

Data for Greenhouse Gas Accounting Procedures in Low Carbon Fuel Policies Overlook the Spatial Variability of Miscanthus-Derived Sustainable Aviation Fuel

Low carbon fuel policies such as the U.S. Renewable Fuel Standard (RFS), Canada Clean Fuel Regulations (CFR), and California Low Carbon Fuel Standard (LCFS) as well as the 45Z tax credit are intended to reduce greenhouse gas (GHG) emissions from transportation. Cellulosic feedstocks, optimized biorefineries, and favorable farming locations can significantly reduce biofuel carbon intensity (CI). Despite advances in field-to-fuel GHG monitoring and flexibility in resource allocation within biorefineries (e.g., governing net electricity production), rigid CI accounting procedures in current policies may limit CI responsiveness across candidate sites and processing facilities. This work examines a hypothetical biomass-to-sustainable aviation fuel (SAF) pathway using miscanthus and alcohol-to-jet (i) to demonstrate how GHG accounting requirements drive estimates of biofuel CIs and (ii) to explore potential CI and financial implications of scenario-specific life cycle assessment (LCA). Results demonstrate that GHG accounting using the CFR/LCFS can reasonably account for distinct levels of net electricity production by a biorefinery, but only the CFR yields similar CI sensitivity to spatially explicit factors (feedstock CI, grid electricity CI) as scenario-specific LCA: most GHG accounting frameworks do not capture CI variation across candidate sites in the United States. Ultimately, this work demonstrates the importance of LCA methodological specifications in low carbon fuel policies and tax credits.

Miscanthus

Optimal Power Management for Large-Scale Battery Energy Storage Systems via Bayesian Inference

Large-scale battery energy storage systems (BESS) have found ever-increasing use across industry and society to accelerate clean energy transition and improve energy supply reliability and resilience. However, their optimal power management poses significant challenges: the underlying high-dimensional nonlinear nonconvex optimization lacks computational tractability in real-world implementation, and the uncertainty of the exogenous power demand makes exact optimization difficult. This paper presents a new solution framework to address these bottlenecks. The solution pivots on introducing power-sharing ratios to specify each cell’s power quota from the output power demand. To find the optimal power-sharing ratios, we formulate a nonlinear model predictive control (NMPC) problem to achieve power-loss-minimizing BESS operation while complying with safety, cell balancing, and power supply-demand constraints. We then propose a parameterized control policy for the power-sharing ratios, which utilizes only three parameters, to reduce the computational demand in solving the NMPC problem. This policy parameterization allows us to translate the NMPC problem into a Bayesian inference problem for the sake of 1) computational tractability, and 2) overcoming the nonconvexity of the optimization problem. We leverage the ensemble Kalman inversion technique to solve the parameter estimation problem. Concurrently, a low-level control loop is developed to seamlessly integrate our proposed approach with the BESS to ensure practical implementation. This low-level controller receives the optimal power-sharing ratios, generates output power references for the cells, and maintains a balance between power supply and demand despite uncertainty in output power. We conduct extensive simulations and experiments on a 20-cell prototype to validate the proposed approach.

Battery energy storage systems (BESSs)

Hybrid renewable energy systems

In the pursuit of ecologically sustainable and resilient energy systems, increasingly more attention is being devoted to a diversity of energy generation and storage methods. As the landscape of generation technology gains nuance and complexity, a wide-ranging set of technical questions has emerged, touching on topics that range from control and optimization of hybrid systems to finance and economic viability to multi-fidelity modeling and scientific machine learning. In the context of this special issue, hybrid renewable energy systems are any systems that consider the combined dynamics of more than one form of generation, storage, or grid subsystem. Research endeavors have delved into improving the flexibility of energy systems by utilizing existing resources, introducing novel operational strategies, deploying enhanced renewable forecasts, and exploring emerging technologies. In conclusion, the interconnection among various sectors has garnered heightened attention, not only due to the provision of additional tradable energy products but also for furnishing flexible headroom to system operators.

29 ENERGY PLANNING, POLICY, AND ECONOMY