Search NASA⌕ Search

SEARCH · Search NASA

Results for “Value function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Massively parallel and universal approximation of nonlinear functions using diffractive processors

Nonlinear computation is essential for a wide range of information processing tasks, yet implementing nonlinear functions using optical systems remains a challenge due to the weak and power-intensive nature of optical nonlinearities. Overcoming this limitation without relying on nonlinear optical materials could unlock unprecedented opportunities for ultrafast and parallel optical computing systems. Here, we demonstrate that large-scale nonlinear computation can be performed using linear optics through optimized diffractive processors composed of passive phase-only surfaces. In this framework, the input variables of nonlinear functions are encoded into the phase of an optical wavefront—e.g., via a spatial light modulator (SLM)—and transformed by an optimized diffractive structure with spatially varying point-spread functions to yield output intensities that approximate a large set of unique nonlinear functions–all in parallel. We provide proof establishing that this architecture serves as a universal function approximator for an arbitrary set of bandlimited nonlinear functions, also covering wavelength-multiplexed nonlinear functions as well as multi-variate and complex-valued functions that are all-optically cascadable. Our analysis also indicates the successful approximation of typical nonlinear activation functions commonly used in neural networks, including the sigmoid, tanh, ReLU (rectified linear unit), and softplus. We numerically demonstrate the parallel computation of one million distinct nonlinear functions, accurately executed at wavelength-scale spatial density at the output of a diffractive optical processor. Furthermore, we experimentally validated this framework using in situ optical learning and approximated 35 unique nonlinear functions in a single shot using a compact setup consisting of an SLM and an image sensor. These results establish diffractive optical processors as a scalable platform for massively parallel universal nonlinear function approximation, paving the way for new capabilities in analog optical computing based on linear materials.

Rahman, Md Sadman Sakib [University of California,↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning

Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.

chance constraints↗
DOE OSTI2024

Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory

Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.

Wang, Yu↗
DOE OSTI2023

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning

We assume that we are given a time series of data from a dynamical system and our task is to learn the flow map of the dynamical system. We present a collection of results on how to enforce constraints coming from the dynamical system in order to accelerate the training of deep neural networks to represent the flow map of the system as well as increase their predictive ability. In particular, we provide ways to enforce constraints during training for all three major modes of learning, namely supervised, unsupervised and reinforcement learning. In general, the dynamic constraints need to include terms which are analogous to memory terms in model reduction formalisms. Such memory terms act as a restoring force which corrects the errors committed by the learned flow map during prediction. For supervised learning, the constraints are added to the objective function. For the case of unsupervised learning, in particular generative adversarial networks, the constraints are introduced by augmenting the input of the discriminator. Finally, for the case of reinforcement learning and in particular actor-critic methods, the constraints are added to the reward function. In addition, for the reinforcement learning case, we present a novel approach based on homotopy of the action-value function in order to stabilize and accelerate training. We use numerical results for the Lorenz system to illustrate the various constructions.

Stinis, Panagiotis↗
DOE OSTI2025

High-order limiting methods using maximum principle bounds derived from the Boltzmann equation I: Euler equations

The use of limiting methods for high-order numerical approximations of hyperbolic conservation laws generally requires defining an admissible region/bounds for the solution. In this work, we present a novel approach for computing solution bounds and limiting for the Euler equations through the kinetic representation provided by the Boltzmann equation, which allows for extending limiters designed for linear advection directly to the Euler equations. Given an arbitrary set of solution values to compute bounds over (e.g., numerical stencil) and a desired linear advection limiter, the proposed approach yields an analytic expression for the admissible region of particle distribution function values, which may be numerically integrated to yield a set of bounds for the density, momentum, and total energy. Further, these solution bounds are shown to preserve positivity of density/pressure/internal energy and, when paired with a limiting technique, can robustly resolve strong discontinuities while recovering high-order accuracy in smooth regions without any ad hoc corrections (e.g., relaxing the bounds). This approach is demonstrated in the context of an explicit unstructured high-order discontinuous Galerkin/flux reconstruction scheme for a variety of difficult problems in gas dynamics, including cases with extreme shocks and shock-vortex interactions. Furthermore, this work presents a foundation for limiting techniques for more complex macroscopic governing equations that can be derived from an underlying kinetic representation for which admissible solution bounds are not well-understood.

42 ENGINEERING↗
DOE OSTI2025

Derivative-free stochastic optimization via adaptive sampling strategies

In this paper, we present a novel derivative-free framework for solving unconstrained stochastic optimization problems. Many problems in fields ranging from simulation optimization to reinforcement learning to quantum computing involve settings where only stochastic function values are obtained via a zeroth-order oracle, which has no available gradient information and necessitates the usage of derivative-free optimization methodologies. Our approach includes estimating gradients using stochastic function evaluations and integrating adaptive sampling techniques to control the accuracy in these stochastic approximations. Our framework encapsulates several gradient estimation techniques, including standard finite-difference, Gaussian smoothing, sphere smoothing, randomized coordinate finite-difference, and randomized subspace finite-difference methods. We provide theoretical convergence guarantees for our framework and analyze the worst-case iteration and sample complexities associated with each gradient estimation method. Finally, we demonstrate the empirical performance of the methods on logistic regression and nonlinear least squares problems.

Adaptive sampling↗
DOE OSTI2020

Primary $\gamma$-ray intensities and $\gamma$-strength functions from discrete two-step $\gamma$-ray cascades in radiative proton-capture experiments

Background: Reaction rates of radiative capture reactions can play a crucial role in the nucleosynthesis of heavy nuclei in explosive stellar environments. These reaction rates depend strongly on $\gamma$-ray decay widths in the reaction products, which are, for nonresonant capture reactions at high excitation energies, derived from the $\gamma$-ray strength function and the nuclear level density. Recently, the ratio method was applied to primary $\gamma$ rays observed from ($\textit{d, p}$) reactions and nuclear resonance fluorescence measurements to extract the dipole strength in atomic nuclei and to test the generalized Brink-Axel hypothesis. Purpose: The purpose of this work is to apply the ratio method to primary $\gamma$-ray intensities of the 63,65 Cu($\textit{p},\gamma$) reactions to extract $\gamma$-ray strength information on the nuclei 64,66 Zn. Here, the impact of spin distribution, total $\gamma$-ray decay widths, level densities, and width fluctuations on the application of the ratio method will be discussed. Additionally, by comparing the relative $\gamma$-ray strength at different excitation energies, conclusions on the validity of the generalized Brink-Axel hypothesis can be made. Method: The radiative proton capture reaction measurements have been performed at the HORUS $\gamma$-ray spectrometer of the University of Cologne at one excitation energy for each reaction. Primary $\gamma$-ray intensities have been determined by normalizing secondary $\gamma$-ray transitions in two-step cascades using their absolute branching ratio. The ratio method was applied to the measured primary $\gamma$-ray intensities as well as to previous measurements by Erlandsson et al. at different excitation energies. Results: The relative strength function curve for 64 Zn from our measurement shows no significant deviation from the previous measurement at a different excitation energy. The same is true for 66 Zn where both measurements were at almost the same excitation energy. Absolute $\gamma$-strength function values have been obtained by normalizing the relative curves to quasiparticle random phase approximation calculations because of the absence of experimental data in the respective energy region. Conclusion: The generalized Brink-Axel hypothesis, i.e., the independence of the strength function on the excitation energy, seems to hold in the studied energy region and nuclei. The method to obtain primary $\gamma$-ray intensities from two-step cascade spectra was shown to be a valuable and sensitive tool although its uncertainties are connected to the knowledge of the low-energy level scheme of the investigated nucleus. The scaling in the ratio method should be taken with care, because the relative strength is not a simple sum of $f_{E1}$ and $f_{M1}$ but a somewhat complex linear combination dependent on the excitation energy of the nucleus.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Memory-efficient emulation of physical tabular data using quadtree decomposition

Computationally expensive functions are sometimes replaced in simulations with an emulator that approx-imates the true function (e.g., equations of state, wavelength-dependent opacity, or composition-dependent materials properties). For functions that have a constrained domain of interest, this can be done by discretizing the domain and performing a local interpolation on the tabulated function values of each local domain. For these so-called tabular data methods, the method of discretizing the domain and mapping the input space to each subdomain can drastically influence the memory and computational costs of the emulator. This is especially true for functions that vary drastically in different regions. We present a method for domain discretization and mapping that utilizes quadtrees, which results in significant reductions in the size of the emulator with minimal increases to computational costs or loss of global accuracy. We apply our method to the electron-positron Helmholtz free energy equation of state and show over an order of magnitude reduction in memory costs for reasonable levels of numerical accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

A Stochastic Quasi-Newton Method in the Absence of Common Random Numbers

We present Q-SASS, a quasi-Newton method for unconstrained stochastic optimization that does not rely on common random numbers. Most existing quasi-Newton approaches leverage common random numbers to construct second-order updates. However, motivated by challenges in variational quantum algorithms—where such coordination is not possible—we consider the setting in which function values and gradients are accessible only through noisy probabilistic zeroth- and first-order oracles, and no common random numbers can be exploited. We derive high-probability tail bounds on the iteration complexity of our algorithm for nonconvex, convex, and strongly convex (more generally, those satisfying the PL condition) objective functions. Finally, we demonstrate the empirical benefits of our quasi-Newton updating scheme on both synthetic and quantum chemistry problems.

Complexity bound↗
DOE OSTI2022

A nonsmooth nonconvex optimization algorithm for two-stage optimization problems

An optimization algorithm for a group of nonsmooth nonconvex problems inspired by two-stage stochastic programming problems is proposed. The main challenges for these problems include (1) the problems lack the popular lower-type properties such as prox-regularity assumed in many nonsmooth nonconvex optimization algorithms, (2) the objective can not be analytically expressed and (3) the evaluation of function values and subgradients are computationally expensive. To address these challenges, this report first examines the properties that exist in many two-stage problems, specifically upper-C 2 objectives. Then, we show that quadratic penalty method for securityconstrained alternating current optimal power flow (SCACOPF) contingency problems can make the contingency solution functions upper-C 2 . Based on these observations, a simplified bundle algorithm that bears similarity to sequential quadratic programming (SQP) method is proposed. It is more efficient in implementation and computation compared to conventional bundle methods. Global convergence analysis of the algorithm is presented under novel and reasonable assumptions. The proposed algorithm therefore fills the gap of theoretical convergence for smoothed SCACOPF problems. The inconsistency that might arise in our treatment of the constraints are addressed through a penalty algorithm whose convergence analysis is also provided. Finally, theoretical capabilities and numerical performance of the algorithm are demonstrated through numerical examples.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Interaction mechanism of transition metal phthalocyanines on transition metal nitride supports

Here, we investigated the electronic interactions between transition metal phthalocyanine (TMPc’s) on a refractory transition metal nitride support, specifically copper phthalocyanine (CuPc) on titanium nitride (TiN). X-ray Photoelectron Spectroscopy (XPS) results suggest a presence of a few nanometer native oxide layer on the surface of the TiN nanoparticles, which consists of TiN, TiO 2 , and Titanium oxynitrides (Ti x O y N z ). A TiNCuPc nanocomposite was synthesized via a simple mixing method due to the strong binding between CuPc and TiN confirmed by density functional theory (DFT) calculations. Both XPS data and DFT calculations revealed an electron transfer from TiN substrate to CuPc molecule. The nature of charge transfer is not influenced by the presence of an oxide layer on the surface of TiN. Substantial deviations are however found between photoelectron emission microscopy (PEEM) measured work function for TiN (4.68 eV) and theoretically calculated work function for pristine stoichiometric TiN (2.63 eV). This behavior is attributed to the presence of an oxide layer on the TiN surface. TiNCuPc composite system has a work function value between those of TiN and CuPc. Our studies open up an opportunity to apply a new class of materials based on transition metal phthalocyanine/transition metal nitride composites to catalysis and optoelectronic devices.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Realizing tunable Fermi level in SnTe by defect control

The tuning of the Fermi level in tin telluride, a topological crystalline insulator, is essential for accessing its unique surface states and optimizing its electronic properties for applications such as spintronics and quantum computing. In this study, we demonstrate that the Fermi level in tin telluride can be effectively modulated by controlling the tin concentration during chemical vapor deposition synthesis. By introducing tin-rich conditions, we observed a blue shift in the x-ray photoelectron spectroscopy core-level peaks of both tin and tellurium, indicating an upward shift in the Fermi level. Further, this shift is corroborated by a decrease in work function values measured via ultraviolet photoelectron spectroscopy, confirming the suppression of Sn vacancies. Our findings provide a low-cost, scalable method to achieve tunable Fermi levels in tin telluride, offering a significant advancement in the development of materials with tailored electronic properties for next-generation technological applications.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2021

Infinite-dimensional fermionic symmetry in supersymmetric gauge theories

We establish the existence of an infinite-dimensional fermionic symmetry in four-dimensional supersymmetric gauge theories by analyzing semiclassical photino dynamics in abelian $\mathscr{N}$ = 1 theories with charged matter. The symmetry is parametrized by a spinor-valued function on an asymptotic S2 at null infinity. It is not manifest at the level of the Lagrangian, but acts non-trivially on physical states, and its Ward identity is the soft photino theorem. The infinite-dimensional fermionic symmetry resides in the same $\mathscr{N}$ = 1 supermultiplet as the physically non-trivial large gauge symmetries associated with the soft photon theorem.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

A matheuristic for design and dispatch of a utility-connected distributed energy system

Modeling distributed power generation systems often requires complicated mathematical expressions that present challenges for commercial optimization solvers. Here, this paper presents a matheuristic to solve a mixed-integer optimization model that informs decisions regarding the design and dispatch of a utility-connected microgrid. We deploy a genetic algorithm to search the system design space and a linear program to solve the economic dispatch problem. The model is a component of a web tool that requires solutions within a few minutes. Our method yields objective function values within 5% of an exogenously produced optimal in fewer than 30 seconds for 90% of our test cases compared to only 10% of our test cases by a traditional optimization solver in the same amount of time.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Be Surface Structures on W(110) and W(211): A DFT Study

Beryllium and Tungsten are promising candidates for use as plasma facing materials (PFMs) in upcoming fusion reactors. Many complex and competing phenomena however complicate the understanding and development of materials that operate in the harsh conditions of the reactor. In particular, redeposition of Be onto the W divertor must be considered due to the expected erosion of the Be first wall under exposure to the plasma. It is known that a build up of Be on W allows for the formation of BeW alloys which can harm the longevity and performance of the fusion divertor. In an effort to understand the interaction of Be with W surfaces, a study of Be structures on W(110) and W(211) as a function of Be coverage has been carried out using Density Functional Theory. We have found that both W surfaces develop monolayers of Be characterized by a densely packed hexagonal structure. Below this monolayer coverage, Be structures are found to have two motifs of coverage, a densely packed hexagonal pattern punctuated by areas of low-density coverage. The structures found here produce work function values and trends in good agreement with experimental measurements.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor-critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection-diffusion field. As a result, we provide explanation for the policy behaviors using knowledge of the underlying physical process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Pareto solutions in multicriteria optimization under uncertainty

We present and analyze several definitions of Pareto optimality for multicriteria optimization or decision problems with uncertainty primarily in their objective function values. In comparison to related notions of Pareto robustness, we first provide a full characterization of an alternative efficient set hierarchy that is based on six different ordering relations both with respect to the multiple objectives and a possibly finite, countably infinite or uncountable number of scenarios. We then establish several scalarization results for the generation of the corresponding efficient points using generalized weighted-sum and epsilon-constraint techniques. Finally, we leverage these scalarization results to also derive more general conditions for the existence of efficient points in each of the corresponding optimality classes, under suitable assumptions.

97 MATHEMATICS AND COMPUTING↗