Search NASA⌕ Search

SEARCH · Search NASA

Results for “Value function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Analytical continuation of matrix-valued functions: Carathéodory formalism

Finite-temperature quantum field theories are formulated in terms of Green's functions and self-energies on the Matsubara axis. In multiorbital systems, these quantities are related to positive semidefinite matrix-valued functions of the Carathéodory and Schur class. Analysis, interpretation, and evaluation of derived quantities such as real-frequency response functions requires analytic continuation of the off-diagonal elements to the real axis. Here we derive the criteria under which such functions exist for given Matsubara data and present an interpolation algorithm that intrinsically respects their mathematical properties. For small systems with precise Matsubara data, we find that the continuation exactly recovers all off-diagonal and diagonal elements. In real-materials systems, we show that the precision of the continuation is sufficient for the analytic continuation to commute with the Dyson equation, and we show that the truncation of the off-diagonal self-energy elements leads to considerable approximation artifacts.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2023

Variational actor-critic algorithms,

We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Artificial intelligence for materials research at extremes

Abstract Materials development is slow and expensive, taking decades from inception to fielding. For materials research at extremes, the situation is even more demanding, as the desired property combinations such as strength and oxidation resistance can have complex interactions. Here, we explore the role of AI and autonomous experimentation (AE) in the process of understanding and developing materials for extreme and coupled environments. AI is important in understanding materials under extremes due to the highly demanding and unique cases these environments represent. Materials are pushed to their limits in ways that, for example, equilibrium phase diagrams cannot describe. Often, multiple physical phenomena compete to determine the material response. Further, validation is often difficult or impossible. AI can help bridge these gaps, providing heuristic but valuable links between materials properties and performance under extreme conditions. We explore the potential advantages of AE along with decision strategies. In particular, we consider the problem of deciding between low-fidelity, inexpensive experiments and high-fidelity, expensive experiments. The cost of experiments is described in terms of the speed and throughput of automated experiments, contrasted with the human resources needed to execute manual experiments. We also consider the cost and benefits of modeling and simulation to further materials understanding, along with characterization of materials under extreme environments in the AE loop. Graphical abstract AI sequential decision-making methods for materials research: Active learning, which focuses on exploration by sampling uncertain regions, Bayesian and bandit optimization as well as reinforcement learning (RL), which trades off exploration of uncertain regions with exploitation of optimum function value. Bayesian and bandit optimization focus on finding the optimal value of the function at each step or cumulatively over the entire steps, respectively, whereas RL considers cumulative value of the labeling function, where the latter can change depending on the state of the system (blue, orange, or green).

36 MATERIALS SCIENCE↗
DOE OSTI2023

Linear-depth quantum circuits for loading Fourier approximations of arbitrary functions

Abstract The ability to efficiently load functions on quantum computers with high fidelity is essential for many quantum algorithms, including those for solving partial differential equations and Monte Carlo estimation. In this work, we introduce the Fourier series loader (FSL) method for preparing quantum states that exactly encode multi-dimensional Fourier series using linear-depth quantum circuits. Specifically, the FSL method prepares a (Dn)-qubit state encoding the 2 Dn -point uniform discretization of aD-dimensional function specified by aD-dimensional Fourier series. A free parameter,m, which must be less thann, determines the number of Fourier coefficients, 2 D ( m + 1 ) , used to represent the function. The FSL method uses a quantum circuit of depth at most 2 ( n − 2 ) + ⌈ log 2 ( n − m ) ⌉ + 2 D ( m + 1 ) + 2 − 2 D ( m + 1 ) , which is linear in the number of Fourier coefficients, and linear in the number of qubits (Dn) despite the fact that the loaded function’s discretization is over exponentially many (2 Dn ) points. The FSL circuit consists of at most D n + 2 D ( m + 1 ) + 1 − 1 single-qubit and D n ( n + 1 ) / 2 + 2 D ( m + 1 ) + 1 − 3 D ( m + 1 ) − 2 two-qubit gates; we present a classical compilation algorithm with runtime O ( 2 3 D ( m + 1 ) ) to determine the FSL circuit for a given Fourier series. The FSL method allows for the highly accurate loading of complex-valued functions that are well-approximated by a Fourier series with finitely many terms. We report results from noiseless quantum circuit simulations, illustrating the capability of the FSL method to load various continuous 1D functions, and a discontinuous 1D function, on 20 qubits with infidelities of less than 10 −6 and 10 −3 , respectively. We also demonstrate the practicality of the FSL method for near-term quantum computers by presenting experiments performed on the Quantinuum H1-1 and H1-2 trapped-ion quantum computers: we loaded a complex-valued function on 3 qubits with a fidelity of over 95 % , as well as various 1D real-valued functions on up to 6 qubits with classical fidelities ≈99%, and a 2D function on 10 qubits with a classical fidelity ≈94%.

Physics↗
DOE OSTI2022

Dynamic, resilient sensing system for automatic cyber-attack neutralization

An industrial asset may have monitoring nodes that generate current monitoring node values. An abnormality detection computer may determine that an abnormal monitoring node is currently being attacked or experiencing fault. A dynamic, resilient estimator constructs, using normal monitoring node values, a latent feature space (of lower dimensionality as compared to a temporal space) associated with latent features. The system also constructs, using normal monitoring node values, functions to project values into the latent feature space. Responsive to an indication that a node is currently being attacked or experiencing fault, the system may compute optimal values of the latent features to minimize a reconstruction error of the nodes not currently being attacked or experiencing a fault. The optimal values may then be projected back into the temporal space to provide estimated values and the current monitoring node values from the abnormal monitoring node are replaced with the estimated values.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Learning Stochastic Parametric Differentiable Predictive Control Policies

We present a scalable unsupervised learning-based method for obtaining explicit control policies for model predictive control problems for stochastic linear systems with additive uncertainties subject to nonlinear chance constraints. We call the proposed method stochastic parametric differentiable predictive control (SP-DPC), which extends the recently proposed deterministic DPC policy optimization algorithm. We formulate the SP-DPC as a deterministic approximation to the stochastic parametric constrained optimal control problem via independent sampling of the problem's parameters and uncertainties. This formulation allows us to directly compute the policy gradients via automatic differentiation of the problem's value function, evaluated over sampled parameters and uncertainties. In particular, the computed expectation of the problem's value function is backpropagated through the finite-time closed-loop system rollouts parametrized by a known nominal system dynamics model and neural control policy. We also provide theoretical probabilistic guarantees on closed-loop stability and chance constraints satisfaction for systems controlled by learned neural policies. We demonstrate the computational efficiency and scalability of the proposed policy optimization algorithm in three numerical examples, including systems with a large number of states or subject to nonlinear constraints.

Drgona, Jan↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint

We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.

data-driven control↗
DOE OSTI2021

Distributionally Robust Partially Observable Markov Decision Process with Moment-Based Ambiguity

In this paper, we consider a distributionally robust partially observable Markov decision process (DR-POMDP), where the distribution of the transition-observation probabilities is unknown at the beginning of each decision period, but their realizations can be inferred using side information at the end of each period after an action being taken. We build an ambiguity set of the joint distribution using bounded moments via conic constraints and seek an optimal policy to maximize the worst-case (minimum) reward for any distribution in the set. We show that the value function of DR-POMDP is piecewise linear convex with respect to the belief state and propose a heuristic search value iteration method for obtaining lower and upper bounds of the value function. We conduct numerical studies and demonstrate the computational performance of our approach via testing instances of a dynamic epidemic control problem. Our results show that DR-POMDP can produce more robust policies under misspecified distributions of transition-observation probabilities as compared to POMDP but has less costly solutions than robust POMDP. The DR-POMDP policies are also insensitive to varying parameter in the ambiguity set and to noise added to the true transition-observation probability values obtained at the end of each decision period.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

On the Sampling-Based Computation of Nash Equilibria Under Uncertainty via the Nikaido–Isoda Function

We consider the computation of an equilibrium of a stochastic Nash equilibrium problem, where the player objectives are assumed to be L 0 -Lipschitz continuous and convex, given rival decisions with convex and closed player-specific feasibility sets. To address this problem, we consider minimizing a suitably defined value function defined using the Nikaido–Isoda function. Such an avenue does not necessitate either monotonicity properties of the concatenated gradient map or potentiality requirements on the game but does require a suitable regularity requirement under which a stationary point is a Nash equilibrium. We design and analyze a sampling-enabled projected-gradient-response method, reliant on inexact resolution of a player-level best-response subproblem. Here, by deriving suitable Lipschitzian guarantees on the value function, we derive both asymptotic guarantees for the sequence of generated iterates as well as rate and complexity guarantees for computing a stationary point by appropriate choices of the sampling rate and inexactness sequence.

Nikaido-Isoda function↗
DOE OSTI2024

Algorithm 1049: The Delaunay Density Diagnostic

Accurate approximation of a real-valued function depends on two aspects of the available data: the density of inputs within the domain of interest and the variation of the outputs over that domain. There are few methods for assessing whether the density of inputs is sufficient to identify the relevant variations in outputs—i.e., the “geometric scale” of the function—despite the fact that sampling density is closely tied to the success or failure of an approximation method. In this article, we introduce a general purpose, computational approach to detecting the geometric scale of real-valued functions over a fixed domain using a deterministic interpolation technique from computational geometry. The algorithm is intended to work on scalar data in moderate dimensions (2–10). Our algorithm is based on the observation that a sequence of piecewise linear interpolants will converge to a continuous function at a quadratic rate (in L 2 norm) if and only if the data are sampled densely enough to distinguish the feature from noise (assuming sufficiently regular sampling). We present numerical experiments demonstrating how our method can identify feature scale, estimate uncertainty in feature scale, and assess the sampling density for fixed (i.e., static) datasets of input–output pairs. Finally, we include analytical results in support of our numerical findings and have released lightweight code that can be adapted for use in a variety of data science settings.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Finite-Time Analysis of Whittle Index based Q-Learning for Restless Multi-Armed Bandits with Neural Network Function Approximation

Whittle index policy is a heuristic to the intractable restless multi-armed bandits (RMAB) problem. Although it is provably asymptotically optimal, finding Whittle indices remains difficult. In this paper, we present Neural-Q-Whittle, a Whittle index based Q-learning algorithm for RMAB with neural network function approximation, which is an example of nonlinear two-timescale stochastic approximation with Q-function values updated on a faster timescale and Whittle indices on a slower timescale. Despite the empirical success of deep Q-learning, the non-asymptotic convergence rate of Neural-Q-Whittle, which couples neural networks with two-timescale Q-learning largely remains unclear. This paper provides a finite-time analysis of Neural-Q-Whittle, where data are generated from a Markov chain, and Q-function is approximated by a ReLU neural network. Our analysis leverages a Lyapunov drift approach to capture the evolution of two coupled parameters, and the nonlinearity in value function approximation further requires us to characterize the approximation error. Combing these provide Neural-Q-Whittle with convergence rate, where is the number of iterations.

reinforcement learning, structured learning, conve↗
DOE OSTI2020

Minimizing total energy cost and tardiness penalty for a scheduling-layout problem in a flexible job shop system: A comparison of four metaheuristic algorithms

Job scheduling and machine layout are interrelated in improving energy consumption (EC) and productivity measures such as tardiness and represent two important decisions that must be made by manufacturers. This interdependency can be explained by transportation time, which connects scheduling and layout. Scheduling and layout, however, have not been thoroughly studied in conjunction using an integrated model in the context of sustainable manufacturing. Hence, we propose an energy-aware optimization model in which scheduling is integrated with layout in a single-level framework. More specifically, a single objective function is defined to minimize the facility energy cost and the job tardiness penalty, which control EC and tardiness respectively in a flexible job shop system. In order to model machine EC more accurately, we also consider three different machine states: a processing state and two idle states. Our case studies show that the integrated model exhibits better performance in controlling manufacturing EC and job tardiness than a non-integrated model in which machine locations are uncontrollable and transportation times between machines are unchangeable. To deal with large-sized problems, we also introduce four new metaheuristics. The performances of these new metaheuristics are compared in terms of objective function values and CPU times using various case studies. The results indicate that a hybrid ant colony optimization and simulated annealing (ACO-SA) algorithm provides better performance than the other algorithms. Specifically, our case studies show that the integrated model using ACO-SA can improve the objective function value by around 5% when compared to the non-integrated model.

42 ENGINEERING↗
DOE OSTI2020

Efficient Ensemble-Based Stochastic Gradient Methods for Optimization Under Geological Uncertainty

Ensemble-based stochastic gradient methods, such as the ensemble optimization (EnOpt) method, the simplex gradient (SG) method, and the stochastic simplex approximate gradient (StoSAG) method, approximate the gradient of an objective function using an ensemble of perturbed control vectors. These methods are increasingly used in solving reservoir optimization problems because they are not only easy to parallelize and couple with any simulator but also computationally more efficient than the conventional finite-difference method for gradient calculations. In this work, we show that EnOpt may fail to achieve sufficient improvement of the objective function when the differences between the objective function values of perturbed control variables and their ensemble mean are large. On the basis of the comparison of EnOpt and SG, we propose a hybrid gradient of EnOpt and SG to save on the computational cost of SG. We also suggest practical ways to reduce the computational cost of EnOpt and StoSAG by approximating the objective function values of unperturbed control variables using the values of perturbed ones. We first demonstrate the performance of our improved ensemble schemes using a benchmark problem. Results show that the proposed gradients saved about 30–50% of the computational cost of the same optimization by using EnOpt, SG, and StoSAG. As a real application, we consider pressure management in carbon storage reservoirs, for which brine extraction wells need to be optimally placed to reduce reservoir pressure buildup while maximizing the net present value. Results show that our improved schemes reduce the computational cost significantly.

58 GEOSCIENCES↗
DOE OSTI2024

An Empirical Quantile Estimation Approach for Chance-Constrained Nonlinear Optimization Problems

We investigate an empirical quantile estimation approach to solve chance-constrained nonlinear optimization problems. Our approach is based on the reformulation of the chance constraint as an equivalent quantile constraint to provide stronger signals on the gradient. In this approach, the value of the quantile function is estimated empirically from samples drawn from the random parameters, and the gradient of the quantile function is estimated via a finite-difference approximation on top of the quantile-function-value estimation. We establish a convergence theory of this approach within the framework of an augmented Lagrangian method for solving general nonlinear constrained optimization problems. The foundation of the convergence analysis is a concentration property of the empirical quantile process, and the analysis is divided based on whether or not the quantile function is differentiable. In contrast to the sampling-and-smoothing approach used in the literature, the method developed in this paper does not involve any smoothing function and hence the quantile-function gradient approximation is easier to implement and there are less accuracy-control parameters to tune. Furthermore, we demonstrate the effectiveness of this approach and compare it with a smoothing method for the quantile-gradient estimation. Numerical investigation shows that the two approaches are competitive for certain problem instances.

Applied Probability↗
DOE OSTI2025

Neural network approaches for parameterized optimal control

Here, we consider numerical approaches for deterministic, finite-dimensional optimal control problems whose dynamics depend on unknown or uncertain parameters. We seek to amortize the solution over a set of relevant parameters in an offline stage to enable rapid decision-making and be able to react to changes in the parameter in the online stage. To tackle the curse of dimensionality arising when the state and/or parameter are high-dimensional, we represent the policy using neural networks. We compare two training paradigms: First, our model-based approach leverages the dynamics and definition of the objective function to learn the value function of the parameterized optimal control problem and obtain the policy using a feedback form. Second, we use actor-critic reinforcement learning to approximate the policy in a data-driven way. Using an example involving a two-dimensional convection-diffusion equation, which features high-dimensional state and parameter spaces, we investigate the accuracy and efficiency of both training paradigms. While both paradigms lead to a reasonable approximation of the policy, the model-based approach is more accurate and considerably reduces the number of PDE solves.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Suppression of Parasitic Voltage Oscillation in Power Modules by Turn-Off Current Zero Placement

This paper presents a frequency-domain analysis of the turn-off equivalent switching network of a hard-switched semiconductor switching cell. This paper also proposes a novel method to suppress parasitic voltage ringing by placing the turn-off current zero at the parasitic resonant frequency. The turn-off current was decomposed into ramp functions and a constant-valued function to facilitate obtaining the frequency components of the turn-off current using the Laplace transform of standard signals. The proposed frequency-domain modeling of the parasitic voltage ringing and the suppression were validated by detailed circuit simulation using parameters corresponding to prior literature with experimental validation.

Galigekere, Veda Prakash↗
DOE OSTI2024

Hydrocarboxylation of C=C Bonds in Polycyclooctene: Progress Toward Valorization of Waste Polyolefins

Upcycling of waste polyolefins into higher-value functional polymers through chemical functionalization is a promising strategy to recover value and minimize their environmental impact. We report the hydrocarboxylation of polycyclooctene as a model for accessing ethylene acrylic acid copolymers (EAA) from waste polyethylene. The effects of various process parameters on the hydrocarboxylation were investigated. The relationship between carboxylic acid concentration and material properties, including crystallinity, melting temperature, glass transition temperature, adhesive properties, and wettability, was studied in comparison to an unfunctionalized polyethylene. Among the catalysts evaluated, Co 2 (CO) 8 exhibited relatively slow kinetics toward the hydrocarboxylation and was not compatible with 2,6-di-tert-butyl hydroxytoluene, a common additive present in commercial polyolefins. PdCl 2 (PPh 3 ) 2 exhibited a higher reactivity toward the hydrocarboxylation, though polymer gelation was observed with extended reaction times or high catalyst loadings. Carboxylic acid incorporation into the polymer was readily controlled by varying the reaction time. Altogether, the resultant COOH-functionalized polyolefins possessed properties analogous to commercial EAA.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Direct evidence of low work function on SrVO 3 cathode using thermionic electron emission microscopy and high-field ultraviolet photoemission spectroscopy

Perovskite SrVO 3 has recently been proposed as a novel electron emission cathode material. Density functional theory (DFT) calculations suggest multiple low work function surfaces, and recent experimental efforts have consistently demonstrated effective work functions of ~2.7 eV for polycrystalline samples, both results suggesting, but not directly confirming, that some fraction of even lower work function surface is present. In this work, thermionic electron emission microscopy (ThEEM) and high-field ultraviolet photoemission spectroscopy (UPS) are used to study the local work function distribution and measure the work function of a partially oriented- (110)-SrVO 3 perovskite oxide cathode surface. Our results show direct evidence of low work function patches of about 2.0 eV on the cathode surface, with a corresponding onset of observable thermionic emission at 750 °C. We hypothesize that, in our ThEEM and UPS experiments, the high applied electric field suppresses the patch field effect, enabling the direct measurement of local work functions. This measured work function of 2.0 eV is comparable to the previous DFT-calculated work function values of the SrVO-terminated (110) SrVO 3 surface (2.3 eV) and SrO-terminated (100) surface (1.9 eV). The measured 2.0 eV value is also much lower than the work function for the (001) LaB 6 single crystal cathode (~2.7 eV) and comparable to the effective work function of B-type dispenser cathodes (~2.1 eV). If SrVO 3 thermionic emitters can be engineered to access domains of this low 2.0 eV work function, they have the potential to significantly improve thermionic emitter-based technologies.

36 MATERIALS SCIENCE↗