Search NASA⌕ Search

SEARCH · Search NASA

Results for “Descent”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Spread Spectrum Time Domain Reflectometry and Steepest Descent Inversion to Measure Complex Impedance

In this paper, we present a method for estimating complex impedances using reflectometry and a modified steepest descent inversion algorithm. We simulate spread spectrum time domain reflectometry (SSTDR), which can measure complex impedances on energized systems for an experimental setup with resistive and capacitive loads. A parametric function, which includes both a misfit function and stabilizer function, is created. The misfit function is a least squares estimate of how close the model data matches observed data. The stabilizer function prevents the steepest descent algorithm from becoming unstable and diverging. Steepest descent iteratively identifies the model parameters that minimize the parametric function. We validate the algorithm by correctly identifying the model parameters (capacitance and resistance) associated with simulated SSTDR data, with added 3 dB white Gaussian noise. With the stabilizer function, the steepest descent algorithm estimates of the model parameters are bounded within a specified range. Furthermore, the errors for capacitance (220pF to 820pF) and resistance (50 Ω to 270 Ω) are < 10%, corresponding to a complex impedance magnitude |R +1/jωC| of 53 Ω to 510 Ω.

14 SOLAR ENERGY↗
DOE OSTI2024

Random coordinate descent: A simple alternative for optimizing parameterized quantum circuits

Variational quantum algorithms rely on the optimization of parameterized quantum circuits in noisy settings. The commonly used back-propagation procedure in classical machine learning is not directly applicable in this setting due to the collapse of quantum states after measurements. Thus, gradient estimations constitute a significant overhead in a gradient-based optimization of such quantum circuits. This paper introduces a random coordinate descent algorithm as a practical and easy-to-implement alternative to the full gradient descent algorithm. This algorithm only requires one partial derivative at each iteration. Motivated by the behavior of measurement noise in the practical optimization of parameterized quantum circuits, this paper presents an optimization problem setting that is amenable to analysis. Under this setting, the random coordinate descent algorithm exhibits the same level of stochastic stability as the full gradient approach, making it as resilient to noise. The complexity of the random coordinate descent method is generally no worse than that of the gradient descent and can be much better for various quantum optimization problems with anisotropic Lipschitz constants. Theoretical analysis and extensive numerical experiments validate our findings. Published by the American Physical Society 2024

Ding, Zhiyan (ORCID:000000018863403X)↗
DOE OSTI2023

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

Abstract We prove, under mild conditions, the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model, both in batch gradient descent and stochastic gradient descent. We also discuss a Riemannian version of the Adam algorithm. We show numerical simulations of these algorithms on various benchmarks.

Whiting, Wes (ORCID:0000000247505060)↗
DOE OSTI2020

Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent

Despite the great achievements of the modern deep neural networks (DNNs), the vulnerability/robustness of state-of-the-art DNNs raises security concerns in many application domains requiring high reliability. Various adversarial attacks are proposed to sabotage the learning performance of DNN models. Among those, the black-box adversarial attack methods have received special attentions owing to their practicality and simplicity. Black-box attacks usually prefer less queries in order to maintain stealthy and low costs. However, most of the current black-box attack methods adopt the first-order gradient descent method, which may come with certain deficiencies such as relatively slow convergence and high sensitivity to hyper-parameter settings. In this paper, we propose a zeroth-order natural gradient descent (ZO-NGD) method to design the adversarial attacks, which incorporates the zeroth-order gradient estimation technique catering to the black-box attack scenario and the second-order natural gradient descent to achieve higher query efficiency. The empirical evaluations on image classification datasets demonstrate that ZO-NGD can obtain significantly lower model query complexities compared with state-of-the-art attack methods.

Zhao, Pu↗
DOE OSTI2020

A Stochastic Gradient Descent Approach for Stochastic Optimal Control

In this work, we introduce a stochastic gradient descent approach to solve the stochastic optimal control problem through stochastic maximum principle. The motivation that drives our method is the gradient of the cost functional in the stochastic optimal control problem is under expectation, and numerical calculation of such an expectation requires fully computation of a system of forward backward stochastic differential equations, which is computationally expensive. By evaluating the expectation with single-sample representation as suggested by the stochastic gradient descent type optimisation, we could save computational efforts in solving FBSDEs and only focus on the optimisation task which aims to determine the optimal control process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Convergence Analysis of a Locally Accelerated Preconditioned Steepest Descent Method for Hermitian-Definite Generalized Eigenvalue Problems

By extending the classical analysis techniques due to Samokish, Faddeev and Faddeeva, and Longsine and McCormick among others, we prove the convergence of preconditioned steepest descent with implicit deflation (PSD-id) method for solving Hermitian-definite generalized eigenvalue problems. Moreover, we derive a nonasymptotic estimate of the rate of convergence of the PSD-id method. We show that with a proper choice of the shift, the indefinite shift-and-invert preconditioner is a locally accelerated preconditioner, and is asymptotically optimal that leads to superlinear convergence. Numerical examples are presented to verify the theoretical results on the convergence behavior of the PSDid method for solving ill-conditioned Hermitian-definite generalized eigenvalue problems arising from electronic structure calculations. While rigorous and full-scale convergence proofs of preconditioned block steepest descent methods in practical use still largely eludes us, we postulate that the theoretical results introduced in this paper sheds light on an improved understanding of the convergence behavior of these block methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

A descent algorithm for generalized complementarity problems based on generalized Fischer-Burmeister functions

We study an unconstrained minimization approach to the generalized complementarity problem GCP(f, g) based on the generalized Fischer-Burmeister function and its generalizations when the underlying functions are C{sup 1}. Also, we show how, under appropriate regularity conditions, minimizing the merit function corresponding to f and g leads to a solution of the generalized complementarity problem. Moreover, we propose a descent algorithm for GCP(f, g) and show a result on the global convergence of a descent algorithm for solving generalized complementarity problem. Finally, we present some preliminary numerical results. Our results further give a unified/generalization treatment of such results for the nonlinear complementarity problem based on generalized Fischer-Burmeister function and its generalizations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

A new hybrid algorithm combining a new chaos optimization approach with gradient descent for high dimensional optimization problems

Chaos optimization algorithm (COA) is a recently developed method for global optimization based on chaos theory. It has the features of easy implementation, short execution time and robust mechanisms of escaping from local optima compared with the existing stochastic searching algorithm. By integrating the gradient descent method with a new COA approach using new strategies including symmetrization and levelling, a novel hybrid method, called SCGD, is proposed in this paper. The main idea is to combine the good ability of exploitation of the gradient descent with the high ability of exploration of COA. The purpose is to avoid being trapped into local optima and to improve convergence in large space and high-dimension optimization problems. The simulation results based on a set of 50 benchmark functions with diverse properties and different dimensions show the efficiency and the abilities of the proposed method in finding the global optima compared with the existing methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Benchmarking optimization methods for materials research: Gradient descent and Bayesian optimization for lithium-ion battery aging diagnostics

Accurate and efficient parameter estimation is essential for battery diagnostics and aging analysis. Here, in this study, we compare two optimization-based approaches—gradient descent and Bayesian optimization—for extracting parameters from differential voltage analysis in lithium-ion batteries. While these techniques are widely used, their relative strengths and limitations for this application are not well understood. The study evaluates the trade-offs between these methods in terms of result quality, computational cost, and reliability within this specific application. The diagnostic results from our battery data suggest adopting gradient descent as an initial method for rapid and efficient analysis, while employing more stable optimization techniques, such as Bayesian optimization, as a verification step to mitigate potential instability. Comparing the two methods provides information on algorithmic choice, while inspiring further discussions on selecting appropriate techniques for specific research tasks.

Zhao, Ziqing [Boston Univ., MA (United States)] (O↗
DOE OSTI2021

Correspondence between neuroevolution and gradient descent

Abstract We show analytically that training a neural network by conditioned stochastic mutation or neuroevolution of its weights is equivalent, in the limit of small mutations, to gradient descent on the loss function in the presence of Gaussian white noise. Averaged over independent realizations of the learning process, neuroevolution is equivalent to gradient descent on the loss function. We use numerical simulation to show that this correspondence can be observed for finite mutations, for shallow and deep neural networks. Our results provide a connection between two families of neural-network training methods that are usually considered to be fundamentally different.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Stochastic gradient descent for optimization for nuclear systems

The use of gradient descent methods for optimizing k-eigenvalue nuclear systems has been shown to be useful in the past, but the use of k-eigenvalue gradients have proved computationally challenging due to their stochastic nature. ADAM is a gradient descent method that accounts for gradients with a stochastic nature. This analysis uses challenge problems constructed to verify if ADAM is a suitable tool to optimize k-eigenvalue nuclear systems. ADAM is able to successfully optimize nuclear systems using the gradients of k-eigenvalue problems despite their stochastic nature and uncertainty. Furthermore, it is clearly demonstrated that low-compute time, high-variance estimates of the gradient lead to better performance in the optimization challenge problems tested here.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

Frequency-domain full-waveform inversion with non-linear descent directions

Full-waveform inversion (FWI) is a highly non-linear inverse problem, normally solved iteratively, with each iteration involving an update constructed through linear operations on the residuals. Incorporating a flexible degree of non-linearity within each update may have important consequences for convergence rates, determination of low model wavenumbers and discrimination of parameters. In this study, we examine one approach for doing so, wherein higher order scattering terms are included within the sensitivity kernel during the construction of the descent direction, adjusting it away from that of the standard Gauss–Newton approach. These scattering terms are naturally admitted when we construct the sensitivity kernel by varying not the current but the to-be-updated model at each iteration. Linear and/or non-linear inverse scattering methodologies allow these additional sensitivity contributions to be computed from the current data residuals within any given update. We show that in the presence of pre-critical reflection data, the error in a second-order non-linear update to a background of s 0 is, in our scheme, proportional to at most (Δs/s 0 ) 3 in the actual parameter jump Δs causing the reflection. In contrast, the error in a standard Gauss–Newton FWI update is proportional to (Δs/s 0 ) 2 . For numerical implementation of more complex cases, we introduce a non-linear frequency-domain scheme, with an inner and an outer loop. A perturbation is determined from the data residuals within the inner loop, and a descent direction based on the resulting non-linear sensitivity kernel is computed in the outer loop. We examine the response of this non-linear FWI using acoustic single-parameter synthetics derived from the Marmousi model. Lastly, the inverted results vary depending on data frequency ranges and initial models, but we conclude that the non-linear FWI has the capability to generate high-resolution model estimates in both shallow and deep regions, and to converge rapidly, relative to a benchmark FWI approach involving the standard gradient.

58 GEOSCIENCES↗
DOE OSTI2018

Random permutations fix a worst case for cyclic coordinate descent

Abstract Variants of the coordinate descent approach for minimizing a nonlinear function are distinguished in part by the order in which coordinates are considered for relaxation. Three common orderings are cyclic (CCD), in which we cycle through the components of $x$ in order; randomized (RCD), in which the component to update is selected randomly and independently at each iteration; and random-permutations cyclic (RPCD), which differs from CCD only in that a random permutation is applied to the variables at the start of each cycle. Known convergence guarantees are weaker for CCD and RPCD than for RCD, though in most practical cases, computational performance is similar among all these variants. There is a certain type of quadratic function for which CCD is significantly slower than for RCD; a recent paper by Sun & Ye (2016, Worst-case complexity of cyclic coordinate descent: $O(n^2)$ gap with randomized version. Technical Report. Stanford, CA: Department of Management Science and Engineering, Stanford University. arXiv:1604.07130) has explored the poor behavior of CCD on functions of this type. The RPCD approach performs well on these functions, even better than RCD in a certain regime. This paper explains the good behavior of RPCD with a tight analysis.

Lee, Ching-pei↗
DOE OSTI2019

Avoiding Communication in Primal and Dual Block Coordinate Descent Methods

Primal and dual block coordinate descent methods are iterative methods for solving regularized and unregularized optimization problems. Distributed-memory parallel implementations of these methods have become popular in analyzing large machine learning datasets. However, existing implementations communicate at every iteration, which, on modern data center and supercomputing architectures, often dominates the cost of floating-point computation. Recent results on communication-avoiding Krylov subspace methods suggest that large speedups are possible by re-organizing iterative algorithms to avoid communication. We show how applying similar algorithmic transformations can lead to primal and dual block coordinate descent methods that only communicate every $s$ iterations---where $s$ is a tuning parameter---instead of every iteration for the regularized least-squares problem. We show that the communication-avoiding variants reduce the number of synchronizations by a factor of $s$ on distributed-memory parallel machines without altering the convergence rate and attain strong scaling speedups of up to $6.1\times$ over the “standard algorithm" on a Cray XC30 supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

CONCURRENT, CONDENSED STEIN VARIATIONAL GRADIENT DESCENT FOR UNCERTAINTY QUANTIFICATION OF NEURAL NETWORKS

In this work, we propose a Stein variational gradient descent (SVGD) method to concurrently sparsify, train, and provide uncertainty quantification (UQ) of a complexly parameterized model, such as a neural network (NN). It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed concurrent, condensed SVGD (ccSVGD) method can provide UQ on parameters, not just outputs. Furthermore, the parameter reduction speeds up the convergence of the Stein gradient descent as it reduces the combinatorial complexity by aligning and differentiating the sensitivity to parameters. These properties are demonstrated with an illustrative example and an application to a mechanical response representation problem in solid mechanics.

42 ENGINEERING↗
DOE OSTI2020

Stochastic gradient descent algorithm for stochastic optimization in solving analytic continuation problems

We propose a stochastic gradient descent based optimization algorithm to solve the analytic continuation problem in which we extract real frequency spectra from imaginary time Quantum Monte Carlo data. The procedure of analytic continuation is an ill-posed inverse problem which is usually solved by regularized optimization methods, such like the Maximum Entropy method, or stochastic optimization methods. The main contribution of this work is to improve the performance of stochastic optimization approaches by introducing a supervised stochastic gradient descent algorithm to solve a flipped inverse system which processes the random solutions obtained by a type of Fast and Efficient Stochastic Optimization Method.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Gradient-Descent-Based Learning Gain for Backstepping Controller and Disturbance Observer of Nonlinear Systems

This paper proposes a gradient-descent-based learning (GL) gain for backstepping controller and disturbance observer (DOB) of nonlinear system. The proposed method consists of the GL gain update law, controller, and DOB. The GL gain update law is proposed to adapt the control gain and DOB gain according to the direction that minimizes the cost function. The mathematical analysis reveals that the GL gain always has a positive sign and upper bound. The controller is designed via a backstepping procedure to track the desired output with GL control gain. The DOB is designed to estimate the unknown external disturbance with the GL DOB gain. Because the control and DOB gains are simultaneously tuned to achieve improved performance, the time consumption for tuning can be reduced. In addition, the peaking phenomenon can be avoided initially by a small initial value of GL gains. The stability of the closed-loop system is guaranteed using the input-to-state stability property. The performance of the proposed method was validated via simulations and experiments using a DC motor.

42 ENGINEERING↗
DOE OSTI2022

State Estimation for Distribution Networks with Asynchronous Sensors Using Stochastic Descent: Preprint

This paper investigates the problem of state estimation for distribution networks with asynchronous sensors comprising of a mix of smart meters and phasor measurement units (PMUs) with multiple sampling and reporting rates. We consider two independent scenarios of state estimation and tracking, with either voltages or currents as states. With these two sets, we investigate estimation under (a) full data, assuming all measurements are available and (b) limited data, where an online algorithmic approach is adopted to estimate the possibly time-varying states by processing measurements as and when available. The proposed algorithm, inspired by the classical Stochastic Gradient Descent (SGD) approach updates the states based on the previous estimate and the newly available measurements. Finally, we demonstrate the estimation and tracking efficacy through numerical simulations on the IEEE-37 test network, while also highlighting how estimation with currents as states leads to faster convergence.

asynchronous sensors↗