Search NASA⌕ Search

SEARCH · Search NASA

Results for “gradient descent optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

143 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparame- ter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For example, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

The Effects of Compounded Model Size Reductions on Adversarial Robustness

Recent advances in Edge AI and Tiny Machine Learning (TinyML) have enabled the deployment of machine learning models on resource-constrained environments. However, deploying these models on edge devices, such as micro-controllers, requires significant model footprint reduction through a variety of techniques such as quantization, pruning, and clustering. While these optimization methods offer considerable advantages, they potentially introduce AI-related security vulnerabilities, particularly concerning model robustness with respect to adversarial AI attacks. Prior research has extensively examined the impact of quantization on adversarial robustness; however, the effects of alternative reduction techniques and their combinations remain understudied. This paper investigates the impact of model size reduction techniques on adversarial robustness, when applied individually and combined. We utilized Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks to generate adversarial perturbations for both training and testing data, and then evaluated the models' accuracy under adversarial training conditions. Our findings revealed that reduction techniques generally diminished robustness; although, combining techniques was not found to make robustness any worse than when applied individually. Moreover, specific techniques can potentially enhance resistance to small size perturbations. This research provides insights into the trade-offs between model size reduction and security, establishing a foundation for future investigations into improving adversarial training techniques and methodologies for maintaining robustness while preserving memory footprint benefits.

Austria, Phillipe [ORNL] (ORCID:0000000236223973)↗
NASA NTRS2025

Hardware in the Loop Performance of Terrestrial Powered Descent Dual Quaternion Guidance With A Custom First-Order Solver

The Safe & Precise Landing Integrated Capabilities Evolution (SPLICE) program continues to push the development of advanced descent and landing technologies. This paper will focus on the improvements made to the SPLICE Dual Quaternion Guidance (DQG) algorithm, which is a critical software component for generating approach phase and hazard avoidance trajectories. The trajectory performance of SPLICE DQG will be evaluated in preparation for a terrestrial flight test that incorporates a hazard avoidance maneuver. Additionally, the implementation of a customized first order Proportional-Integral Projected Gradient solver will be reviewed, along with preliminary execution results on the SPLICE Descent and Landing Computer (DLC).

SPLICE↗
DOE OSTI2025

Closed-Form Approximation of the Total Variation Proximal Operator

Total variation (TV) is a widely used function for regularizing imaging inverse problems that is particularly appropriate for images whose underlying structure is piecewise constant. TV regularized optimization problems are typically solved using proximal methods, but the way in which they are applied is constrained by the absence of a closed-form expression for the proximal operator of the TV function. A closed-form approximation of the TV proximal operator has previously been proposed, but its accuracy was not theoretically explored in detail. Here, we address this gap by making several new theoretical contributions, proving that the approximation leads to a proximal operator of some convex function, it is equivalent to a gradient descent step on a smoothed version of TV, and that its error can be fully characterized and controlled with its scaling parameter. We experimentally validate our theoretical results on image denoising and sparse-view computed tomography (CT) image reconstruction.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Qualitative and quantitative enhancement of parameter estimation for model-based diagnostics using automatic differentiation with an application to inertial fusion

Parameter estimation using observables is a fundamental concept in the experimental sciences. Mathematical models that represent the physical processes can enable reconstructions of the experimental observables and greatly assist in parameter estimation by turning it into an optimization problem which can be solved by gradient-free or gradient-based methods. In this work, the recent rise in flexible frameworks for developing differentiable scientific computing programs is leveraged in order to dramatically accelerate data analysis of a common experimental diagnostic relevant to laser–plasma and inertial fusion experiments, Thomson scattering. A differentiable Thomson-scattering data analysis tool is developed that uses reverse-mode automatic differentiation (AD) to calculate gradients. By switching from finite differencing to reverse-mode AD, three distinct outcomes are achieved. First, gradient descent is accelerated dramatically to the extent that it enables near real-time usage in laser–plasma experiments. Second, qualitatively novel quantities which require $\mathcal{O}(10^3)$ parameters can now be included in the analysis of data which enables unprecedented measurements of small-scale laser–plasma phenomena. Third, uncertainty estimation approaches that leverage the value of the Hessian become accurate and efficient because reverse-mode AD can be used for calculating the Hessian.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Bias-Variance Trade-Off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs

Physics-Informed Neural Networks (PINNs) have triggered a paradigm shift in scientific computing, leveraging mesh-free properties and robust approximation capabilities. While proving effective for low-dimensional partial differential equations (PDEs), the computational cost of PINNs remains a hurdle in high-dimensional scenarios. This is particularly pronounced when computing high-order and high-dimensional derivatives in the physics-informed loss. Randomized Smoothing PINN (RS-PINN) introduces Gaussian noise for stochastic smoothing of the original neural net model, enabling the use of Monte Carlo methods for derivative approximation, which eliminates the need for costly automatic differentiation. Despite its computational efficiency, especially in the approximation of high-dimensional derivatives, RS-PINN introduces biases in both loss and gradients, negatively impacting convergence, especially when coupled with stochastic gradient descent (SGD) algorithms. We present a comprehensive analysis of biases in RS-PINN, attributing them to the nonlinearity of the Mean Squared Error (MSE) loss as well as the intrinsic nonlinearity of the PDE itself. We propose tailored bias correction techniques, delineating their application based on the order of PDE nonlinearity. The derivation of an unbiased RS-PINN allows for a detailed examination of its advantages and disadvantages compared to the biased version. Specifically, the biased version has a lower variance and runs faster than the unbiased version, but it is less accurate due to the bias. To optimize the bias-variance trade-off, we combine the two approaches in a hybrid method that balances the rapid convergence of the biased version with the high accuracy of the unbiased version. In addition to methodological contributions, we present an enhanced implementation of RS-PINN. Extensive experiments on diverse high-dimensional PDEs, including Fokker-Planck, Hamilton-Jacobi-Bellman (HJB), viscous Burgers’, Allen-Cahn, and Sine-Gordon equations, illustrate the bias-variance trade-off and highlight the effectiveness of the hybrid RS-PINN. Empirical guidelines are provided for selecting biased, unbiased, or hybrid versions, depending on the dimensionality and nonlinearity of the specific PDE problem.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Efficient ptychographic phase retrieval via a matrix-free Levenberg-Marquardt algorithm

The phase retrieval problem, where one aims to recover a complex-valued image from far-field intensity measurements, is a classic problem encountered in a range of imaging applications. Modern phase retrieval approaches usually rely on gradient descent methods in a nonlinear minimization framework. Calculating closed-form gradients for use in these methods is tedious work, and formulating second order derivatives is even more laborious. Additionally, second order techniques often require the storage and inversion of large matrices of partial derivatives, with memory requirements that can be prohibitive for data-rich imaging modalities. We use a reverse-mode automatic differentiation (AD) framework to implement an efficient matrix-free version of the Levenberg-Marquardt (LM) algorithm, a longstanding method that finds popular use in nonlinear least-square minimization problems but which has seen little use in phase retrieval. Furthermore, we extend the basic LM algorithm so that it can be applied for more general constrained optimization problems (including phase retrieval problems) beyond just the least-square applications. Since we use AD, we only need to specify the physics-based forward model for a specific imaging application; the first and second-order derivative terms are calculated automatically through matrix-vector products, without explicitly forming the large Jacobian or Gauss-Newton matrices typically required for the LM method. We demonstrate that this algorithm can be used to solve both the unconstrained ptychographic object retrieval problem and the constrained “blind” ptychographic object and probe retrieval problems, under the popular Gaussian noise model as well as the Poisson noise model. We compare this algorithm to state-of-the-art first order ptychographic reconstruction methods to demonstrate empirically that this method outperforms best-in-class first-order methods: it provides excellent convergence guarantees with (in many cases) a superlinear rate of convergence, all with a computational cost comparable to, or lower than, the tested first-order algorithms.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Dynamic calibration of differential equations using machine learning, with application to turbulence models

We present a methodology for calibration of parametric ordinary and partial differential equation models, using off-the-shelf software for back-propagation in Neural Networks (NN). As a prototypical example, we consider calibration of a Reynolds-averaged Navier-Stokes (RANS) turbulence closure model, against ground truth data from direct numerical simulations (DNS) of two different turbulent flows. Numerical time integration is represented as a custom NN, where only the RANS model parameters are trainable. A loss function is defined to quantify the mismatch between the NN prediction and the ground truth over a predefined, finite time integration window. This loss function is then minimized using a gradient descent method utilizing the back-propagation algorithm. Furthermore, this dynamic approach to training is to be contrasted with a static approach, wherein a least square regression estimate for parameters is obtained in the limit of an infinitesimal time integration window. In a first test of static and dynamic approaches against ground truth data generated by the model, the former proves to be significantly faster and more accurate than the latter at recovering the parameters. When both calibration approaches are tested against DNS data, for which it is known that the model cannot achieve a perfect fit, the static approach yields a good prediction only for short times, while the dynamic approach results in physical and stable predictions over the entire integration window. After optimization of the dynamic approach for time step, spatial resolution, stability, and physics-based constraints, we obtain a 50% improvement of outcomes over those obtained from the existing, manually calibrated set of parameters, demonstrating the merits of this systematic and automated procedure.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1985

Optimal landing of a helicopter in autorotation

Gliding descent in autorotation is a maneuver used by helicopter pilots in case of engine failure. The landing of a helicopter in autorotation is formulated as a nonlinear optimal control problem. The OH-58A helicopter was used. Helicopter vertical and horizontal velocities, vertical and horizontal displacement, and the rotor angle speed were modeled. An empirical approximation for the induced veloctiy in the vortex-ring state were provided. The cost function of the optimal control problem is a weighted sum of the squared horizontal and vertical components of the helicopter velocity at touchdown. Optimal trajectories are calculated for entry conditions well within the horizontal-vertical restriction curve, with the helicopter initially in hover or forwared flight. The resultant two-point boundary value problem with path equality constraints was successfully solved using the Sequential Gradient Restoration Technique.

Lee, A. Y. N.↗
DOE OSTI2022

A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks

To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (Biometrika 81(3):425–455, 1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform simulated and real data Monte Carlo experiments to quantify the effectiveness of our approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

An Online Dynamic Amplitude-Correcting Gradient Estimation Technique to Align X-ray Focusing Optics

High-brightness X-ray pulses, as generated at synchrotrons and X-ray free electron lasers (XFELs), are used in a variety of scientific experiments. At these facilities, measurements often require optical equipment, e.g Compound Refractive Lenses (CRLs) to be precisely aligned and focused. The lateral alignment of CRLs to a beamline requires precise positioning along four axes: two translational, and the two rotational. At a synchrotron, alignment is often accomplished manually. However, XFEL beamlines present a beam brightness that fluctuates stochastically, making manual alignment a time-consuming endeavor. Automation using simplex or classic stochastic descent often fails, given the errant gradient estimates. Herein we present a dynamic-amplitude correction to the usual gradient based on the combination of a generalized finite difference stencil and a time-dependent sampling pattern. Intensity is recorded periodically, then used to normalize numerical derivatives against fluctuations. Error expectation is analyzed, and efficacy is demonstrated on classic benchmarks. We provide a proof of concept by laterally aligning optics on a simulated XFEL beamline using data recorded at both synchrotron and XFEL facilities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima.

machine leaning, Bad minima, Inverse problems, Phy↗
DOE OSTI2022

Reverse-mode differentiation in arbitrary tensor network format: with application to supervised learning.

This paper describes an efficient reverse-mode differentiation algorithm for contraction operations of tensor networks that may have arbitrary and unconventional network topologies. The approach leverages the tensor contraction tree of Evenbly and Pfeifer (2014), which provides an instruction set for the contraction sequence of a network. We show that this tree can be efficiently leveraged for differentiation of a full tensor network contraction using a recursive scheme that exploits (1) the bilinear property of contraction and (2) the property that trees have single path from root to leaves. While differentiation of tensor-tensor contraction is already possible in most automatic differentiation packages, we show that exploiting these two additional properties in the specific context of contraction sequences can improve efficiency. Following a description of the algorithm and computational complexity analysis, we investigate its utility for gradient-based supervised learning for low-rank function recovery and for fitting real-world unstructured datasets. We demonstrate improved performance over alternating least-squares optimization approaches and the capability to handle heterogeneous and arbitrary tensor network formats. When compared to alternating minimization algorithms, we find that the gradient-based approach requires a smaller oversampling ratio (number of samples compared to number model parameters) for recovery. This increased efficiency extends to fitting unstructured data of varying dimensionality and when employing a variety of tensor network formats. Here, we show improved learning using the hierarchical Tucker method over the tensor-train in high-dimensional settings on a number of benchmark problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Prediction of gas hydrate saturation using machine learning and optimal set of well-logs

We report resistivity and acoustic logs are widely used to estimate gas hydrate saturation in various sedimentary systems using one of the two popular methods ((1) acoustic velocity and (2) electrical resistivity), but the limitations of these two methods are often overlooked, which include (i) well-specific calibration of empirical exponents in the electrical resistivity method, (ii) assumption of known pore morphology for gas hydrates in the acoustic velocity method, and (iii) presence of unknown mineralogy and bulk modulus terms in the acoustic velocity method. NMR-density porosity-derived gas hydrate saturation based on the analysis of the transverse magnetization relaxation time (T2) is considered the most precise method, but acquisition of NMR-based logs is limited at relatively recent drilled sites; additionally, its use in conventional oil and gas reservoirs is not that common due to higher cost and operational deployment limitations associated with acquiring NMR well-logs. This study proposes a new method that predicts gas hydrate saturation (S h ) for any well using porosity, bulk density, and compressional wave (P wave) velocity well-logs with neural network (or stochastic gradient descent regression) without any well-specific calibration and/or other aforementioned shortcomings of the existing methods. The method is developed by examining the underlying dependency between S h and different combinations of well-logs, chosen from 6 routine logs, with 12 different machine learning (ML) algorithms. The accuracy of the proposed method in predicting S h is ~ 84%, which is better than the accuracy of seismic and electrical resistivity methods (≤ 75%) per the results reported by three different studies. The robustness of the method in the specific case of permafrost-associated gas hydrates is demonstrated with well-log data from two wells drilled on the Alaska North Slope.

58 GEOSCIENCES↗
DOE OSTI2024

Convergence of variational Monte Carlo simulation and scale-invariant pre-training

We provide theoretical convergence bounds for the variational Monte Carlo (VMC) method as applied to optimize neural network wave functions for the electronic structure problem. Here, we study both the energy minimization phase and the supervised pre-training phase that is commonly used prior to energy minimization. For the energy minimization phase, the standard algorithm is scale-invariant by design, and we provide a proof of convergence for this algorithm without modifications. The pre-training stage typically does not feature such scale-invariance. We propose using a scale-invariant loss for the pretraining phase and demonstrate empirically that it leads to faster pre-training.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Training neural networks using Metropolis Monte Carlo and an adaptive variant

Abstract We examine the zero-temperature Metropolis Monte Carlo (MC) algorithm as a tool for training a neural network by minimizing a loss function. We find that, as expected on theoretical grounds and shown empirically by other authors, Metropolis MC can train a neural net with an accuracy comparable to that of gradient descent (GD), if not necessarily as quickly. The Metropolis algorithm does not fail automatically when the number of parameters of a neural network is large. It can fail when a neural network’s structure or neuron activations are strongly heterogenous, and we introduce an adaptive Monte Carlo algorithm (aMC) to overcome these limitations. The intrinsic stochasticity and numerical stability of the MC method allow aMC to train deep neural networks and recurrent neural networks in which the gradient is too small or too large to allow training by GD. MC methods offer a complement to gradient-based methods for training neural networks, allowing access to a distinct set of network architectures and principles.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Progress on Inverse Estimation Technique of Non-Linear Pitch Damping Coefficient Curves Using Free-Flight CFD Generated Trajectories

Characterization of entry vehicle pitch damping coefficient curves is crucial to ensure appropriate re-entry and overall mission success. The pitch damping coefficient (C_(m_q )+C_(m_α ̇ )) is used to encapsulate the oscillatory growth or decay of a body during a trajectory. The inverse estimation technique utilizes an existing Free-Flight CFD (FF-CFD) dataset and wraps a reconstruction algorithm in an optimizer. The reconstruction integrates the planar equations of motion derived by Schoenenberger, Queen [1] using Python’s scipy.integrate.solve_ivp. The optimizer’s objective function is the normalized 𝐿2 residual of the angle of attack peaks between the reconstructed trajectory and the original data produced with FF-CFD. Inclusion of the peak times in this residual calculation allows for simultaneous optimization of the pitch moment coefficient, C_(m_α ). This residual equation is shown below in Eq. 1. The optimizer scipy.optimize.minimize was used with the gradient-based Powell method for the analysis presented, however the differential evolution method was investigated as means of comparison, and was found to produce marginally lower residual values with prohibitively longer run times. Further, the pitch damping curve is found by fitting a cubic interpolation function to a set of (α, (C_(m_q )+C_(m_α ̇ ))) control points, where the α points are held constant and the (C_(m_q )+C_(m_α ̇ )) values are the optimized parameters. The pitch moment curve uses a linear interpolation between the minimum and maximum α in the dataset. FF-CFD generated trajectories using the Dragonfly capsule geometry with the Genesis ballistic range model parameters were simulated and used for this analysis. These FF-CFD trajectories simulate planar motion, as restricted by the reconstructing the equations of motion, of three different cases: 1-DoF (free-to-pitch), 2-DoF (free-to-pitch and heave), and 3-DoF (free-to-pitch, heave, and decelerate). Pitch damping coefficient curves generated using this inverse estimation curve technique with FF-CFD 1-DoF Dragonfly data are found in Fig. 1. Preliminary results reconstructing ballistic range shots using these FF-CFD derived predictions of the pitch damping curve (Fig. 1) are shown in Fig. 2. It should be noted that the ballistic range shot used a Genesis model whereas the FF-CFD data used a Dragonfly geometry, however these geometries are similar.

entry↗