Search NASA⌕ Search

SEARCH · Search NASA

Results for “gradient descent”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Accelerating gradient descent and Adam via fractional gradients

Here we propose a class of novel fractional-order optimization algorithms. We define a fractional-order gradient via the Caputo fractional derivatives that generalizes integer-order gradient. We refer it to as the Caputo fractional-based gradient, and develop an efficient implementation to compute it. A general class of fractional-order optimization methods is then obtained by replacing integer-order gradients with the Caputo fractional-based gradients. To give concrete algorithms, we consider gradient descent (GD) and Adam, and extend them to the Caputo fractional GD (CfGD) and the Caputo fractional Adam (CfAdam). We demonstrate the superiority of CfGD and CfAdam on several large scale optimization problems that arise from scientific machine learning applications, such as ill-conditioned least squares problem on real-world data and the training of neural networks involving non-convex objective functions. Numerical examples show that both CfGD and CfAdam result in acceleration over GD and Adam, respectively. We also derive error bounds of CfGD for quadratic functions, which further indicate that CfGD could mitigate the dependence on the condition number in the rate of convergence and results in significant acceleration over GD.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Accelerated Sparse Recovery via Gradient Descent with Nonlinear Conjugate Gradient Momentum

This paper applies an idea of adaptive momentum for the nonlinear conjugate gradient to accelerate optimization problems in sparse recovery. Specifically, we consider two types of minimization problems: a (single) differentiable function and the sum of a non-smooth function and a differentiable function. In the first case, we adopt a fixed step size to avoid the traditional line search and establish the convergence analysis of the proposed algorithm for a quadratic problem. This acceleration is further incorporated with an operator splitting technique to deal with the non-smooth function in the second case. As a result, we use the convex ι 1 and the nonconvex ι 1 – ι 2 functionals as two case studies to demonstrate the efficiency of the proposed approaches over traditional methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

Abstract We prove, under mild conditions, the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model, both in batch gradient descent and stochastic gradient descent. We also discuss a Riemannian version of the Adam algorithm. We show numerical simulations of these algorithms on various benchmarks.

Whiting, Wes (ORCID:0000000247505060)↗
DOE OSTI2023

Stochastic gradient descent for optimization for nuclear systems

The use of gradient descent methods for optimizing k-eigenvalue nuclear systems has been shown to be useful in the past, but the use of k-eigenvalue gradients have proved computationally challenging due to their stochastic nature. ADAM is a gradient descent method that accounts for gradients with a stochastic nature. This analysis uses challenge problems constructed to verify if ADAM is a suitable tool to optimize k-eigenvalue nuclear systems. ADAM is able to successfully optimize nuclear systems using the gradients of k-eigenvalue problems despite their stochastic nature and uncertainty. Furthermore, it is clearly demonstrated that low-compute time, high-variance estimates of the gradient lead to better performance in the optimization challenge problems tested here.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

A Stochastic Gradient Descent Approach for Stochastic Optimal Control

In this work, we introduce a stochastic gradient descent approach to solve the stochastic optimal control problem through stochastic maximum principle. The motivation that drives our method is the gradient of the cost functional in the stochastic optimal control problem is under expectation, and numerical calculation of such an expectation requires fully computation of a system of forward backward stochastic differential equations, which is computationally expensive. By evaluating the expectation with single-sample representation as suggested by the stochastic gradient descent type optimisation, we could save computational efforts in solving FBSDEs and only focus on the optimisation task which aims to determine the optimal control process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Stochastic gradient descent algorithm for stochastic optimization in solving analytic continuation problems

We propose a stochastic gradient descent based optimization algorithm to solve the analytic continuation problem in which we extract real frequency spectra from imaginary time Quantum Monte Carlo data. The procedure of analytic continuation is an ill-posed inverse problem which is usually solved by regularized optimization methods, such like the Maximum Entropy method, or stochastic optimization methods. The main contribution of this work is to improve the performance of stochastic optimization approaches by introducing a supervised stochastic gradient descent algorithm to solve a flipped inverse system which processes the random solutions obtained by a type of Fast and Efficient Stochastic Optimization Method.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

CONCURRENT, CONDENSED STEIN VARIATIONAL GRADIENT DESCENT FOR UNCERTAINTY QUANTIFICATION OF NEURAL NETWORKS

In this work, we propose a Stein variational gradient descent (SVGD) method to concurrently sparsify, train, and provide uncertainty quantification (UQ) of a complexly parameterized model, such as a neural network (NN). It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed concurrent, condensed SVGD (ccSVGD) method can provide UQ on parameters, not just outputs. Furthermore, the parameter reduction speeds up the convergence of the Stein gradient descent as it reduces the combinatorial complexity by aligning and differentiating the sensitivity to parameters. These properties are demonstrated with an illustrative example and an application to a mechanical response representation problem in solid mechanics.

42 ENGINEERING↗
DOE OSTI2020

Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent

Despite the great achievements of the modern deep neural networks (DNNs), the vulnerability/robustness of state-of-the-art DNNs raises security concerns in many application domains requiring high reliability. Various adversarial attacks are proposed to sabotage the learning performance of DNN models. Among those, the black-box adversarial attack methods have received special attentions owing to their practicality and simplicity. Black-box attacks usually prefer less queries in order to maintain stealthy and low costs. However, most of the current black-box attack methods adopt the first-order gradient descent method, which may come with certain deficiencies such as relatively slow convergence and high sensitivity to hyper-parameter settings. In this paper, we propose a zeroth-order natural gradient descent (ZO-NGD) method to design the adversarial attacks, which incorporates the zeroth-order gradient estimation technique catering to the black-box attack scenario and the second-order natural gradient descent to achieve higher query efficiency. The empirical evaluations on image classification datasets demonstrate that ZO-NGD can obtain significantly lower model query complexities compared with state-of-the-art attack methods.

Zhao, Pu↗
DOE OSTI2021

Correspondence between neuroevolution and gradient descent

Abstract We show analytically that training a neural network by conditioned stochastic mutation or neuroevolution of its weights is equivalent, in the limit of small mutations, to gradient descent on the loss function in the presence of Gaussian white noise. Averaged over independent realizations of the learning process, neuroevolution is equivalent to gradient descent on the loss function. We use numerical simulation to show that this correspondence can be observed for finite mutations, for shallow and deep neural networks. Our results provide a connection between two families of neural-network training methods that are usually considered to be fundamentally different.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2021

Multi-Pass Sequential Mini-Batch Stochastic Gradient Descent Algorithms for Noise Covariance Estimation in Adaptive Kalman Filtering

Estimation of unknown noise covariances in a Kalman filter is a problem of significant practical interest in a wide array of applications. Although this problem has a long history, reliable algorithms for their estimation were scant, and necessary and sufficient conditions for identifiability of the covariances were in dispute until recently. Necessary and sufficient conditions for covariance estimation and a batch estimation algorithm were presented in our previous study. This paper presents stochastic gradient descent algorithms for noise covariance estimation in adaptive Kalman filters that are an order of magnitude faster than the batch method for similar or better root mean square error. More significantly, these algorithms are applicable to non-stationary systems where the noise covariances can occasionally jump up or down by an unknown magnitude. The computational efficiency of the new algorithms stems from adaptive thresholds for convergence, recursive fading memory estimation of the sample cross-correlations of the innovations, and accelerated stochastic gradient descent algorithms. The comparative evaluation of the proposed methods on a number of test cases demonstrates their computational efficiency and accuracy.

Adaptive Kalman filtering↗
DOE OSTI2025

Benchmarking optimization methods for materials research: Gradient descent and Bayesian optimization for lithium-ion battery aging diagnostics

Accurate and efficient parameter estimation is essential for battery diagnostics and aging analysis. Here, in this study, we compare two optimization-based approaches—gradient descent and Bayesian optimization—for extracting parameters from differential voltage analysis in lithium-ion batteries. While these techniques are widely used, their relative strengths and limitations for this application are not well understood. The study evaluates the trade-offs between these methods in terms of result quality, computational cost, and reliability within this specific application. The diagnostic results from our battery data suggest adopting gradient descent as an initial method for rapid and efficient analysis, while employing more stable optimization techniques, such as Bayesian optimization, as a verification step to mitigate potential instability. Comparing the two methods provides information on algorithmic choice, while inspiring further discussions on selecting appropriate techniques for specific research tasks.

Zhao, Ziqing [Boston Univ., MA (United States)] (O↗
DOE OSTI2023

Gradient-Descent-Based Learning Gain for Backstepping Controller and Disturbance Observer of Nonlinear Systems

This paper proposes a gradient-descent-based learning (GL) gain for backstepping controller and disturbance observer (DOB) of nonlinear system. The proposed method consists of the GL gain update law, controller, and DOB. The GL gain update law is proposed to adapt the control gain and DOB gain according to the direction that minimizes the cost function. The mathematical analysis reveals that the GL gain always has a positive sign and upper bound. The controller is designed via a backstepping procedure to track the desired output with GL control gain. The DOB is designed to estimate the unknown external disturbance with the GL DOB gain. Because the control and DOB gains are simultaneously tuned to achieve improved performance, the time consumption for tuning can be reduced. In addition, the peaking phenomenon can be avoided initially by a small initial value of GL gains. The stability of the closed-loop system is guaranteed using the input-to-state stability property. The performance of the proposed method was validated via simulations and experiments using a DC motor.

42 ENGINEERING↗
DOE OSTI2025

Convergence analysis for a nonlocal gradient descent method via directional Gaussian smoothing

We analyze the convergence of a nonlocal gradient descent method for minimizing a class of high-dimensional non-convex functions, where a directional Gaussian smoothing (DGS) is proposed to define the nonlocal gradient (also referred to as the DGS gradient). The method was first proposed in [Zhang et al., Enabling long-range exploration in minimization of multimodal functions, UAI 2021], in which multiple numerical experiments showed that replacing the traditional local gradient with the DGS gradient can help the optimizers escape local minima more easily and significantly improve their performance. However, a rigorous theory for the efficiency of the method on nonconvex landscape is lacking. In this work, we investigate the scenario where the objective function is composed of a convex function, perturbed by deterministic oscillating noise. We provide a convergence theory under which the iterates exponentially converge to a tightened neighborhood of the solution, whose size is characterized by the noise wavelength. Here, we also establish a correlation between the optimal values of the Gaussian smoothing radius and the noise wavelength, thus justifying the advantage of using moderate or large smoothing radii with the method. Furthermore, if the noise level decays to zero when approaching the global minimum, we prove that DGS-based optimization converges to the exact global minimum with linear rates, similarly to standard gradient-based methods in optimizing convex functions. Several numerical experiments are provided to confirm our theory and illustrate the superiority of the approach over those based on the local gradient.

Tran, Hoang [Oak Ridge National Laboratory (ORNL),↗
DOE OSTI2025

Enhancing ACPF Analysis: Integrating Newton-Raphson Method with Gradient Descent and Computational Graphs

This paper presents a new method for enhancing Alternating Current Power Flow (ACPF) analysis. The method integrates the Newton-Raphson (NR) method with Enhanced-Gradient Descent (GD) and computational graphs. The integration of renewable energy sources in power systems introduces variability and unpredictability, and this method addresses these challenges. It leverages the robustness of NR for accurate approximations and the flexibility of GD for handling variable conditions, all without requiring Jacobian matrix inversion. Furthermore, computational graphs provide a structured and visual framework that simplifies and systematizes the application of these methods. The goal of this fusion is to overcome the limitations of traditional ACPF methods and improve the resilience, adaptability, and efficiency of modern power grid analyses. We validate the effectiveness of our advanced algorithm through comprehensive testing on established IEEE benchmark systems. Furthermore, our findings demonstrate that our approach not only speeds up the convergence process but also ensures consistent performance across diverse system states, representing a significant advancement in power flow computation.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Plateau Phenomenon in Gradient Descent Training of RELU Networks: Explanation, Quantification, and Avoidance

The ability of neural networks to provide ‘best in class’ approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods,a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon.analysis is carried out in the setting of univariate ReLU networks. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. Here, the main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Implementation of Stochastic Gradient Descent in an Automated Glow Peak Identification Software for Multiple Thermoluminescent Dosimeter Types

A glow-curve analysis code was previously developed in C++ to analyze thermoluminescent dosimeter glow curves using automated peak detection while applying a first-order kinetics model. A newer version of this code was implemented to improve the automated peak detection and curve fitting models. The Stochastic Gradient Descent Algorithm was introduced to replace the prior approach of taking first and second-order derivatives for peak detection. Additionally, early stopping mechanisms were invoked to improve the previously used Levenberg-Marquardt Algorithm employed for curve fitting. The two software versions were compared through glow curve analysis of different thermoluminescent dosimeter materials and calculation of the corresponding figures of merit. Altogether improvements were shown, namely an increase in the number of peaks detected and a reduction of the mean figure of merit by approximately 46%.

137Cs↗
DOE OSTI2020

A Nonlocal-Gradient Descent Method for Inverse Design in Nanophotonics

Local-gradient-based optimization approaches lack nonlocal exploration abilityrequired for escaping from local minima when searching non-convex landscapes.A directional Gaussian smoothing (DGS) approach was recently proposed in [29]and used to define a truly nonlocal gradient, referred to as the DGS gradient, inorder to enable nonlocal exploration in high-dimensional black-box optimization.Promising results show that replacing the traditional local gradient with the nonlocalDGS gradient can significantly improve the performance of gradient-based methodsin optimizing highly multi-modal loss functions. However, the current DGS methodis designed for unbounded and uncontrained optimization problems, making itinapplicable to real-world engineering optimization problems where the tuningparameters are often bounded and the loss function is usually constrained byphysical processes. In this work, we propose to extend to the DGS approachto the constrained inverse design framework in order to find better optima ofmulti-modal loss functions. A series of adaptive strategies for smoothing radiusand learning rate updating are developed to improve the computational efficiencyand robustness. Our methodology is demonstrated by an example of designing ananoscale wavelength demultiplexer, and shows superior performance compared tothe state-of-the-art approaches. By incorporating volume constraints, the optimizeddesign achieves an equivalently high performance but significantly reduces theamount of material usage.

Bi, Sirui↗
DOE OSTI2023

Training material models using gradient descent algorithms

High temperature design requires accurate constitutive models to describe material inelastic deformation and failure behavior. Oftentimes, calibrating accurate models devolves into the problem of fitting the model parameters against experimental test data. Here, we present the pyopmat package, an open source framework for calibrating constitutive models against experiment data subjected to various loading conditions using machine learning techniques. The package calculates the exact gradient of the model response with respect to the parameters using a combination of automatic differentiation and the adjoint method. Given this exact gradient, we compare the performance of several gradient-based optimization techniques in fitting realistic constitutive models against data. Here, we demonstrate the efficiency and accuracy of our package through example problems using both synthetic data, generated using known parameter sets, under monotonic and cyclic loading conditions and also with an example applying the techniques developed here to actual high temperature creep-fatigue test data.

36 MATERIALS SCIENCE↗