Search NASA⌕ Search

SEARCH · Search NASA

Results for “, policy gradient”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search

In this paper, we develop a grid-interactive multi-zone building controller based on a deep reinforcement learning (RL) approach. The controller is designed to facilitate building operation during normal conditions and demand response events, while ensuring occupants comfort and energy efficiency. We leverage a continuous action space RL formulation, and devise a two-stage global-local RL training framework. In the first stage, a global fast policy search is performed using a gradient-free RL algorithm. In the second stage, a local fine-tuning is conducted using a policy gradient method. In contrast to the state-of-the-art model predictive control (MPC) approach, the proposed RL controller does not require complex computation during real-time operation and can adapt to nonlinear building models. We illustrate the controller performance numerically using a five-zone commercial building.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2021

Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search: Preprint

In this paper, we develop a grid-interactive multi-zone building controller based on a deep reinforcement learning (RL) approach. The controller is designed to facilitate building operation during normal conditions and demand response events, while ensuring occupants comfort and energy efficiency. We leverage a continuous action space RL formulation, and devise a two-stage global-local RL training framework. In the first stage, a global fast policy search is performed using a gradient-free RL algorithm. In the second stage, a local fine-tuning is conducted using a policy gradient method. In contrast to the state-of-the-art model predictive control (MPC) approach, the proposed RL controller does not require complex computation during real-time operation and can adapt to non-linear building models. We illustrate the controller performance numerically using a five-zone commercial building.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2022

Stochastic Learning Approach for Binary Optimization: Application to Bayesian Optimal Design of Experiments

Here, we present a novel stochastic approach to binary optimization suited for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The probabilistic objective is then solved by using a stochastic optimization routine to find an optimal observational policy. This formulation (a) is generally applicable to binary optimization problems with soft constraints and is ideal for OED and sensor placement problems; (b) does not require differentiability of the original objective function (e.g., a utility function in OED applications) with respect to the design variable, and thus it enables direct employment of sparsity-enforcing penalty functions such as $\ell_0$, without needing to utilize a continuation procedure or apply a rounding technique; (c) exhibits much lower computational cost than traditional gradient-based relaxation approaches; and (d) can be applied to both linear and nonlinear OED problems with proper choice of the utility function. The proposed approach is analyzed from an optimization perspective with detailed convergence analysis of the optimization approach and is also analyzed from a machine learning perspective with correspondence to policy gradient reinforcement learning. The approach is demonstrated numerically by using an idealized two-dimensional Bayesian linear inverse problem and validated by extensive numerical experiments carried out for sensor placement in a parameter identification setup.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Variational actor-critic algorithms,

We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Multi-Agent Safe Policy Learning for Power Management of Networked Microgrids

This paper presents a supervised multi-agent safe policy learning (SMAS-PL) method for optimal power management of networked microgrids (MGs) in distribution systems. While unconstrained reinforcement learning (RL) algorithms are black-box decision models that could fail to satisfy grid operational constraints, our proposed method considers AC power flow equations and other operational limits. Accordingly, the training process employs the gradient information of operational constraints to ensure that the optimal control policy functions generate safe and feasible decisions. Furthermore, we have developed a distributed consensus-based optimization approach to train the agents’ policy functions while maintaining MGs’ privacy and data ownership boundaries. After training, the learned optimal policy functions can be safely used by the MGs to dispatch their local resources, without the need to solve a complex optimization problem from scratch. Lastly, numerical experiments have been devised to verify the performance of the proposed method.

, policy gradient↗
DOE OSTI2021

Analysis of Residential Time-of-Use Utility Rate Structures and Economic Implications for Thermal Energy Storage

Thermal energy storage (TES) is an increasingly popular tool to level out the daily electrical demand and add stability to the electrical grid as more intermittent renewable energy sources are installed. TES systems can locally decouple high thermal loads from the operation of a heat pump or reduce the electrical energy demand of the heat pump by providing a more favorable temperature gradient. In addition, many policy makers and utility providers have introduced time-of-use (TOU) rate schedules for residential customers to better reflect the price of electricity generation and demand for specific times. TOU rate schedules price grid-provided electricity differently throughout the day depending on the region’s climate, time of year, and electrical production. Large differences between on-peak and off-peak electrical prices may create an economic advantage for a residential customer to install a TES system. In this work, the economic and energy savings are calculated for a modeled 2,400 square foot residential building with water/ice-based TES using a TOU rate structure. The weather data is from Fresno County, CA, ASHRAE climate zone 3B, and a representative residential TOU utility rate structure from Pacific Gas and Electric (PG&E) was used. The results showed that total energy consumption could be reduced by 14.5% with an 87.5% reduction in on-peak energy usage when the TES is installed. The cost of operating this system for space cooling was reduced by nearly 20% using the sample utility rate plan.

Sultan, Sara↗
DOE OSTI2022

Two-Stage Reinforcement Learning Policy Search for Grid-Interactive Building Control

This paper develops an intelligent grid-interactive building controller, which optimizes building operation during both normal hours and demand response (DR) events. To avoid costly on-demand computation and to adapt to non-linear building models, the controller utilizes reinforcement learning (RL) and makes real-time decisions based on a near-optimal control policy. Learning such a policy typically amounts to solving a hard non-convex optimization problem. We propose to address this problem with a novel global-local policy search method. In the first stage, an RL algorithm based on zero-order gradient estimation is leveraged to search for the optimal policy globally, due to its scalability and the potential to escape some poor performing local optima. The obtained policy is then fine-tuned locally to bring the first-stage solution closer to that of the original unsmoothed problem. Experiments on a simulated five-zone commercial building demonstrate the advantages of the proposed method over existing learning approaches. They also show that the learned control policy outperforms a pragmatic linear model predictive controller (MPC) and approaches the performance of an oracle MPC in testing scenarios. Using a state-of-the-art advanced computing system, we demonstrate that the controller can be learned and deployed within hours of training.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Binary Quantum Control Optimization with Uncertain Hamiltonians

Optimizing the controls of quantum systems plays a crucial role in advancing quantum technologies. The time-varying noises in quantum systems and the widespread use of inhomogeneous quantum ensembles raise the need for high-quality quantum controls under uncertainties. In this paper, we consider a stochastic discrete optimization formulation of a discretized binary optimal quantum control problem involving Hamiltonians with predictable uncertainties. We propose a sample-based reformulation that optimizes both risk-neutral and risk-averse measurements of control policies, and solve these with two gradient-based algorithms using sum-up-rounding approaches. Furthermore, we discuss the differentiability of the objective function and prove upper bounds of the gaps between the optimal solutions to binary control problems and their continuous relaxations. We conduct numerical simulations on various sized problem instances based on two applications of quantum pulse optimization; we evaluate different strategies to mitigate the impact of uncertainties in quantum systems. In conclusion, we demonstrate that the controls of our stochastic optimization model achieve significantly higher quality and robustness compared with the controls of a deterministic model.

conditional value-at-risk (CVaR)↗
DOE OSTI2022

Climate Warming Alters Nutrient Storage in Seasonally Dry Forests: Insights From a 2,300 m Elevation Gradient

Understanding potential response of forest carbon (C) and nutrient storage to warming is important for climate mitigation policies. Unfortunately, those responses are difficult to predict in seasonally dry forests, in part, because ecosystem processes are highly sensitive to both changes in temperature and precipitation. We investigated how warming might alter stocks of C, nitrogen (N), and phosphorus (P) in vegetation and the entire regolith (soil + weathered bedrock or “saprock”) using a space-for-time substitution along a bioclimatic gradient in the Sierra Nevada, California. The pine-oak and mixed-conifer forests between 1,160–2,015 m elevation have more optimal climates (not too dry or hot) for ecosystem productivity, soil weathering, and cycling of essential elements than the oak savannah (405 m) and subalpine forest (2,700 m). We found decreases in overstory vegetation nutrient stocks with decreasing elevation because of enhanced water limitation and greater occurrence of disturbances. Stocks of C, N, and P in the entire regolith peaked at the pine-oak and mixed-conifer forests across the bioclimatic gradient, driven by thicker regolith profiles and greater nutrient input rates. These observations suggest long-term warming will decrease ecosystem nutrient storage at the warmer, transitional pine-oak zone, but will increase nutrient storage at the colder, subalpine zone. Assuming steady-state conditions, we found the mean residence time of ecosystem C decreased with projected rising air temperatures and increased following a major drought event across the bioclimatic gradient. Our study emphasizes potentially elevation-dependent changes in nutrient storage and C persistence with warming in seasonally dry forests.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Unsupervised Resource Allocation with Graph Neural Networks

We present an approach for maximizing a global utility function by learning how to allocate resources in an unsupervised way. We expect interactions between allocation targets to be important and therefore propose to learn the reward structure for near-optimal allocation policies with a GNN. By relaxing the resource constraint, we can employ gradient-based optimization in contrast to more standard evolutionary algorithms. Our algorithm is motivated by a problem in modern astronomy, where one needs to select-based on limited initial information-among $10^9$ galaxies those whose detailed measurement will lead to optimal inference of the composition of the universe. Our technique presents a way of flexibly learning an allocation strategy by only requiring forward simulators for the physics of interest and the measurement process. We anticipate that our technique will also find applications in a range of resource allocation problems.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

From Optimization to Sampling Through Gradient Flows

Optimization and sampling algorithms play a central role in science and engineering as they enable finding optimal predictions, policies, and recommendations, as well as expected and equilibrium states of complex systems. The notion of “optimality” is formalized by the choice of an objective function, while the notion of an “expected” state is specified by a probabilistic model for the distribution of states. Optimizing rugged objective functions and sampling multimodal distributions is computationally challenging, especially in high-dimensional problems. Here, for this reason, many optimization and sampling methods have been developed by researchers working in disparate fields such as Bayesian statistics, molecular dynamics, genetics, quantum chemistry, machine learning, weather forecasting, econometrics, and medical imaging.

Trillos, N. García↗
DOE OSTI2022

Extracting energy from ocean thermal and salinity gradients to power unmanned underwater vehicles: State of the art, current limitations, and future outlook

Thermal gradient energy-generation technologies for powering unmanned underwater vehicles (UUVs) or autonomous sensing systems in the ocean are mainly in the research development phase or commercially available at a limited scale, and salinity-gradient energy-generation technologies have not been adequately researched yet. The demand for self-powered UUVs suitable for long-term deployments has been growing, and further research related to small-scale ocean gradient energy systems is needed. In this study, we conducted a comprehensive review about harvesting energy from ocean thermal or salinity gradients for powering UUVs, focusing on gliders and profiling floats. Thermal gradient energy systems for UUVs based on phase change materials (PCM) cannot provide the energy required for powering autonomous sensing systems because of the systems' low energy conversion efficiency. Besides reducing energy consumption by developing more efficient electrical-mechanical systems, enhancing the thermal conductivity of the PCMs may help address this challenge by increasing the power generation rate of the UUVs. Several other emerging technologies, such as thermoelectric generators, shape memory alloys, and small-scale thermodynamic cycle systems, have shown potential for powering UUVs, but they are still only at the laboratory testing or conceptual design phase. The most advanced power generation technologies based on salinity gradients, reverse electrodialysis and pressure-retarded osmosis, are still not economically viable for large-scale deployment, mainly because of the high cost of the components required to operate in harsh saline environments. Our feasibility evaluation showed that existing salinity gradient power generation technologies are not directly feasible for powering UUVs in the open ocean.

16 TIDAL AND WAVE POWER↗
DOE OSTI2025

A Two-Stage Quantum Reinforcement Learning Method for Multi-Objective Transmission Switching

Multi-objective transmission switching (MO-TS) problems involve the strategic reconfiguration of network topology to simultaneously optimize multiple objectives. As the system scale increases, finding feasible solutions becomes increasingly challenging due to the problem's nonlinearity and high computational complexity. To address these challenges, this paper proposes a two-stage quantum reinforcement learning method that leverages potential quantum advantages for MO-TS. In the first stage, candidate switching lines are identified using a graph-theoretical approach to reduce the problem's dimensionality. The second stage introduces a quantum-classical reinforcement learning framework, where a learnable measurement-based CNN-ResVQC architecture is developed to effectively reduce the input dimension for quantum processing, mitigate vanishing gradients, and enhance trainability while improving the quantum circuit's flexibility in modeling complex decision policies for MO-TS. Numerical studies on IEEE 14-bus, 57-bus, and 118-bus systems demonstrate that the proposed algorithm achieves superior training stability and faster convergence with approximately 1% of the network parameters required by classical algorithms, highlighting its effectiveness, efficiency, and scalability. Furthermore, the practicality is validated through its stable convergence under three common quantum noise channels.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2021

Dairy Methane Emissions in California's San Joaquin Valley Inferred With Ground‐Based Remote Sensing Observations in the Summer and Winter

Abstract The dairy industry in the San Joaquin Valley (SJV) is one of California’s largest methane (CH 4 ) sources. Reducing dairy emissions is a priority for the state’s climate change plans. Observations of current dairy CH 4 emissions are key to monitoring actions taken toward this goal. To help support this, we present new ground‐based measurements of atmospheric column‐averaged CH 4 mixing ratio (XCH 4 ) gradients across a group of 600 dairies in the central SJV using EM27/SUN solar spectrometers. We used measurements from the 2019 summer and 2020 winter seasons for a top‐down emission inversion based on the WRF‐STILT model. Our top‐down estimates of the region’s dairy emissions range from 90% to 183% of the current CALGEM inventory’s emissions of 277 Gg/yr. In contrast to the strong temperature dependence found by earlier dairy CH 4 emission studies, we also find that our top‐down emissions during the winter measurement days are comparable to the summer measurement days, possibly due to seasonal changes in dairy management practices and meteorological conditions. Furthermore, we find significant interday variability in our measurements and find that our emission estimates overlap with earlier top‐down studies and bottom‐up inventories in this region. Our study demonstrates how analysis of ground‐based remotely sensed CH 4 gradient observations can help improve our understanding of CH 4 sources at scales relevant to mitigation policy. It also reflects the need for long‐term monitoring of CH 4 emissions in the region and at individual facilities to better understand their emissions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Method and apparatus for constructing informative outcomes to guide multi-policy decision making

In Multi-Policy Decision-Making (MPDM), many computationally-expensive forward simulations are performed in order to predict the performance of a set of candidate policies. In risk-aware formulations of MPDM, only the worst outcomes affect the decision making process, and efficiently finding these influential outcomes becomes the core challenge. Recently, stochastic gradient optimization algorithms, using a heuristic function, were shown to be significantly superior to random sampling. In this disclosure, it was shown that accurate gradients can be computed-even through a complex forward simulation—using approaches similar to those in dep networks. The proposed approach finds influential outcomes more reliably, and is faster than earlier methods, allowing one to evaluate more policies while simultaneously eliminating the need to design an easily-differentiable heuristic function.

Olson, Edwin↗
DOE OSTI2025

Hyperspectral leaf reflectance of grasses varies with evolutionary lineage more than with site

Abstract To predict ecological responses at broad environmental scales, grass species are commonly grouped into two broad functional types based on photosynthetic pathway. However, closely related species may have distinctive anatomical and physiological attributes that influence ecological responses, beyond those related to photosynthetic pathway alone. Hyperspectral leaf reflectance can provide an integrated measure of covarying leaf traits that may result from phylogenetic trait conservatism and/or environmental conditions. Understanding whether spectra‐trait relationships are lineage specific or reflect environmental variation across sites is necessary for using hyperspectral reflectance to predict plant responses to environmental changes across spatial scales. We measured hyperspectral leaf reflectance (400–2400 nm) and 12 structural, biochemical, and physiological leaf traits from five grass‐dominated sites spanning the Great Plains of North America. We assessed if variation in leaf reflectance spectra among grass species is explained more by evolutionary lineage (as captured by tribes or subfamilies), photosynthetic pathway (C 3 or C 4 ), or site differences. We then determined whether leaf spectra can be used to predict leaf traits within and across lineages. Our results using redundancy analysis ordination (RDA) show that grass tribe identity explained more variation in leaf spectra (adjusted R 2 = 0.12) than photosynthetic pathway, which explained little variation in leaf spectra (adjusted R 2 = 0.00). Furthermore, leaf reflectance from the same tribe across multiple sites was more similar than leaf reflectance from the same site across tribes (adjusted R 2 = 0.12 and 0.08, respectively). Across all sites and species, trait predictions based on spectra ranged considerably in predictive accuracies ( R 2 = 0.65 to <0.01), but R 2 was >0.80 for certain lineages and sites. The relationship between Vc max , a measure of photosynthetic capacity, and spectra was particularly promising. Chloridoideae, a lineage more common at drier sites, appears to have distinct spectra‐trait relationships compared with other lineages. Overall, our results show that evolutionary relatedness explains more variation in grass leaf spectra than photosynthetic pathway or site, but consideration of lineage‐ and site‐specific trait relationships is needed to interpret spectral variation across large environmental gradients.

Pau, Stephanie [Department of Geography University↗
DOE OSTI2026

IntraShuffler: A Privacy Preserving Framework for Heterogeneous DP Federated Learning

Heterogeneous Differential Privacy (HDP) in Federated Learning (FL) allows clients to select individual privacy budgets () according to institutional policies and data sensitivity. In practice, many HDP-FL systems employ -aware server aggregation to improve model utility by re-weighting client updates according to their declared privacy budgets. However, gradient updates in FL retain structural patterns induced by non-independent and identically-distributed (non-IID) data, and these additional signals exposed by -aware aggregation create new opportunities for inference by an honest-but-curious server. In this work, we first show that a server equipped with gradient denoising and surrogate modeling can mount a Privacy Inference Attack that infers distributional attributes of clients and links updates from the same client across training rounds, measured via surrogate inference accuracy and linkage success, under realistic knowledge constraints. The Shuffle-Model has been widely studied as a defense against such inference risks by anonymizing update sources, but it is fundamentally incompatible with HDP-FL -aware aggregation. To address this challenge, we propose IntraShuffler, a middleware defense framework designed for HDP-FL systems. IntraShuffler introduces a privacy-aware shuffling mechanism that groups clients into privacy-compatible buckets and performs parameter-level shuffling within each bucket to disrupt persistent gradient structure while preserving -aware aggregation. Experiments across four different datasets show that IntraShuffler reduces gradient recoverability by over 60% and decreases surrogate inference accuracy from 0.78 to 0.33 while maintaining comparable model utility across multiple FL aggregation rules.

Riya, Farhin Farhad [ORNL]↗