Search NASA⌕ Search

SEARCH · Search NASA

Results for “Hessian approximation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

An adaptive Hessian approximated stochastic gradient MCMC method

Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their ability to handle large datasets and the potential to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. Here, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation (SA) to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the samples and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix via SA, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and demonstrates better learning results compared to standard SG-MCMC updating rules. The approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. Numerical experiments are performed on several problems, including sampling from 2D correlated distribution, synthetic regression problems, and learning the numerical solutions of heterogeneous elliptic PDE. The numerical results demonstrate great improvement in both the convergence rate and accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Approximate Hessian for accelerating ab initio structure relaxation by force fitting

We present a method to approximate the Hessian matrix of the Born-Oppenheimer energy landscape by using a simple force field model whose parameters are fitted to on-the-flight ab initio results. The inversed Hessian matrix is used as the preconditioner of conjugate gradient algorithms to speed up the atomic structure relaxation, resulting in a speedup factor of 2 to 5 on systems of bulk, slab, sheets, and atomic clusters. Because the force field model employed is simple and general, the parameter fitting is straightforward; the method is applicable to a variety of complicated systems for minimum structure relaxation. In the metal cluster new structure search, the new method yields better structures than the one obtained before with conventional algorithms.

36 MATERIALS SCIENCE↗
DOE OSTI2011

Fast alogorithms for Bayesian uncertainty quantification in large-scale linear inverse problems based on low-rank partial Hessian approximations

We consider the problem of estimating the uncertainty in large-scale linear statistical inverse problems with high-dimensional parameter spaces within the framework of Bayesian inference. When the noise and prior probability densities are Gaussian, the solution to the inverse problem is also Gaussian, and is thus characterized by the mean and covariance matrix of the posterior probability density. Unfortunately, explicitly computing the posterior covariance matrix requires as many forward solutions as there are parameters, and is thus prohibitive when the forward problem is expensive and the parameter dimension is large. However, for many ill-posed inverse problems, the Hessian matrix of the data misfit term has a spectrum that collapses rapidly to zero. We present a fast method for computation of an approximation to the posterior covariance that exploits the lowrank structure of the preconditioned (by the prior covariance) Hessian of the data misfit. Analysis of an infinite-dimensional model convection-diffusion problem, and numerical experiments on large-scale 3D convection-diffusion inverse problems with up to 1.5 million parameters, demonstrate that the number of forward PDE solves required for an accurate low-rank approximation is independent of the problem dimension. This permits scalable estimation of the uncertainty in large-scale ill-posed linear inverse problems at a small multiple (independent of the problem dimension) of the cost of solving the forward problem.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Hierarchical off-diagonal low-rank approximation of Hessians in inverse problems, with application to ice sheet model initialization

Obtaining lightweight and accurate approximations of discretized objective functional Hessians in inverse problems governed by partial differential equations (PDEs) is essential to make both deterministic and Bayesian statistical large-scale inverse problems computationally tractable. The cubic computational complexity of dense linear algebraic tasks, such as Cholesky factorization, that provide a means to sample Gaussian distributions and determine solutions of Newton linear systems is a computational bottleneck at large-scale. These tasks can be reduced to log-linear complexity by utilizing hierarchical off-diagonal low-rank (HODLR) matrix approximations. In this work, we show that a class of Hessians that arise from inverse problems governed by PDEs are well approximated by the HODLR matrix format. In particular, we study inverse problems governed by PDEs that model the instantaneous viscous flow of ice sheets. In these problems, we seek a spatially distributed basal sliding parameter field such that the flow predicted by the ice sheet model is consistent with ice sheet surface velocity observations. Here, we demonstrate the use of HODLR Hessian approximation to efficiently sample the Laplace approximation of the posterior distribution with covariance further approximated by HODLR matrix compression. Computational studies are performed which illustrate ice sheet problem regimes for which the Gauss–Newton data-misfit Hessian is more efficiently approximated by the HODLR matrix format than the low-rank (LR) format. We then demonstrate that HODLR approximations can be favorable, when compared to global LR approximations, for large-scale problems by studying the data-misfit Hessian associated with inverse problems governed by the first-order Stokes flow model on the Humboldt glacier and Greenland ice sheet.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

A randomized sketching trust-region secant method for low-memory dynamic optimization

The numerical solution of dynamic optimization problems is often limited by the memory required to store the state trajectory, which is used to evaluate the objective function and its derivatives. Recently, [R. Muthukumar et al., SIAM Journal on Optimization 31(2), pp. 1242–1275 (2021)] introduced a trust-region method for dynamic optimization that employs randomized sketching to compress the state trajectory, resulting in inexact derivative computations. By adaptively learning the sketch rank, the trust-region algorithm achieves rigorous convergence guarantees. Here, we extend this approach to use secant Hessian approximations. Due to the randomness introduced by the sketch, the traditional secant update formulae can produce poor Hessian approximations. In particular, the difference of two gradients, computed from two different sketches, may be inconsistent. To overcome this, we employ a sketched approximation of the Hessian application, in lieu of computing the gradient difference. We numerically demonstrate the improved stability of this approach on an example from PDE-constrained optimization.

dynamic optimization↗
NASA NTRS1998

Accelerated Training for Large Feedforward Neural Networks

In this paper we introduce a new training algorithm, the scaled variable metric (SVM) method. Our approach attempts to increase the convergence rate of the modified variable metric method. It is also combined with the RBackprop algorithm, which computes the product of the matrix of second derivatives (Hessian) with an arbitrary vector. The RBackprop method allows us to avoid computationally expensive, direct line searches. In addition, it can be utilized in the new, 'predictive' updating technique of the inverse Hessian approximation. We have used directional slope testing to adjust the step size and found that this strategy works exceptionally well in conjunction with the Rbackprop algorithm. Some supplementary, but nevertheless important enhancements to the basic training scheme such as improved setting of a scaling factor for the variable metric update and computationally more efficient procedure for updating the inverse Hessian approximation are presented as well. We summarize by comparing the SVM method with four first- and second- order optimization algorithms including a very effective implementation of the Levenberg-Marquardt method. Our tests indicate promising computational speed gains of the new training technique, particularly for large feedforward networks, i.e., for problems where the training process may be the most laborious.

Stepniewski, Slawomir W.↗
NASA NTRS1987

Recursive inverse kinematics for robot arms via Kalman filtering and Bryson-Frazier smoothing

This paper applies linear filtering and smoothing theory to solve recursively the inverse kinematics problem for serial multilink manipulators. This problem is to find a set of joint angles that achieve a prescribed tip position and/or orientation. A widely applicable numerical search solution is presented. The approach finds the minimum of a generalized distance between the desired and the actual manipulator tip position and/or orientation. Both a first-order steepest-descent gradient search and a second-order Newton-Raphson search are developed. The optimal relaxation factor required for the steepest descent method is computed recursively using an outward/inward procedure similar to those used typically for recursive inverse dynamics calculations. The second-order search requires evaluation of a gradient and an approximate Hessian. A Gauss-Markov approach is used to approximate the Hessian matrix in terms of products of first-order derivatives. This matrix is inverted recursively using a two-stage process of inward Kalman filtering followed by outward smoothing. This two-stage process is analogous to that recently developed by the author to solve by means of spatial filtering and smoothing the forward dynamics problem for serial manipulators.

Rodriguez, G.↗
DOE OSTI2014

A finite difference Davidson procedure to sidestep full ab initio hessian calculation: Application to characterization of stationary points and transition state searches

The cost of calculating nuclear hessians, either analytically or by finite difference methods, during the course of quantum chemical analyses can be prohibitive for systems containing hundreds of atoms. In many applications, though, only a few eigenvalues and eigenvectors, and not the full hessian, are required. For instance, the lowest one or two eigenvalues of the full hessian are sufficient to characterize a stationary point as a minimum or a transition state (TS), respectively. We describe here a method that can eliminate the need for hessian calculations for both the characterization of stationary points as well as searches for saddle points. A finite differences implementation of the Davidson method that uses only first derivatives of the energy to calculate the lowest eigenvalues and eigenvectors of the hessian is discussed. This method can be implemented in conjunction with geometry optimization methods such as partitioned-rational function optimization (P-RFO) to characterize stationary points on the potential energy surface. With equal ease, it can be combined with interpolation methods that determine TS guess structures, such as the freezing string method, to generate approximate hessian matrices in lieu of full hessians as input to P-RFO for TS optimization. This approach is shown to achieve significant cost savings relative to exact hessian calculation when applied to both stationary point characterization as well as TS optimization. The basic reason is that the present approach scales one power of system size lower since the rate of convergence is approximately independent of the size of the system. Therefore, the finite-difference Davidson method is a viable alternative to full hessian calculation for stationary point characterization and TS search particularly when analytical hessians are not available or require substantial computational effort.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2014

Using Hessian update formulae to construct modified Shepard interpolated potential energy surfaces: Application to vibrating surface atoms

Modified Shepard interpolation based on second order Taylor series expansions has proven to be a flexible tool for constructing potential energy surfaces in a range of situations. Extending this to gas–surface dynamics where surface atoms are allowed to move represents a substantial increase in the dimensionality of the problem, reflected in a dramatic increase in the computational cost of the required Hessian (matrix of second derivatives) evaluations. This work demonstrates that using approximate Hessians derived from well known Hessian update formulae and a single accurate Hessian can provide an effective way to avoid this expensive accurate Hessian determination.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2019

A memory-distributed quasi-Newton solver for nonlinear programming problems with a small number of general constraints

We approach the problem of parallelizing state-of-the-art nonlinear programming optimization algorithms.Specifically, we focus on parallelizing quasi-Newton interior-point methods that use limited-memory secant Hessian approximations. Such interior-point methods are known to have better convergence properties and to be more effective on large-scale problems than gradient-based and derivative-free optimization algorithms. We target nonlinear and potentially nonconvex optimization problems with an arbitrary number of bound constraints and a small number of general equality and inequality constraints on the optimization variables. These problems occur for example in the form of optimal control, optimal design, and inverse problems governed by ordinary or partial differential equations, whenever they are expressed in a “reduced-space” optimization approach. We introduce and analyze the time and space complexity of a decomposition method for solving the quasi-Newton linear systems that leverages the fact that the quasi-Newton Hessian matrix has a small number of dense blocks that border a low-rank update of a diagonal matrix. This enables an efficient parallelization on memory-distributed computers of the iterations of the optimization algorithm, a state-of-the-art filter line-search interior-point algorithm by Wächter et. al. We illustrate the efficiency of the proposed method by solving structural topology optimization problems on up to 4608 cores on a parallel machine.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2012

Fast compressed sensing-based CBCT reconstruction using Barzilai-Borwein formulation for application to on-line IGRT

Purpose: Compressed sensing theory has enabled an accurate, low-dose cone-beam computed tomography (CBCT) reconstruction using a minimal number of noisy projections. However, the reconstruction time remains a significant challenge for practical implementation in the clinic. In this work, we propose a novel gradient projection algorithm, based on the Gradient-Projection-Barzilai-Borwein formulation (GP-BB), that handles the total variation (TV)-norm regularization-based least squares problem for the CBCT reconstruction in a highly efficient manner, with speed acceptable for routine use in the clinic. Methods: CBCT is reconstructed by minimizing an energy function consisting of a data fidelity term and a TV-norm regularization term. Both terms are simultaneously minimized by calculating the gradient projection of the energy function with the step size determined using an approximate Hessian calculation at each iteration, based on the Barzilai-Borwein formulation. To speed up the process, a multiresolution optimization is used. In addition, the entire algorithm was designed to run with a single graphics processing unit (GPU) card. To evaluate the performance, the Shepp-Logan numerical phantom, the CatPhan 600 physical phantom, and a clinically-treated head-and-neck patient were acquired from the TrueBeam system (Varian Medical Systems, Palo Alto, CA). For each scan, in total, 364 projections were acquired in a 200 deg. rotation. The imager has 1024 x 768 pixels with 0.388 x 0.388-mm resolution. This was down-sampled to 512 x 384 pixels with 0.776 x 0.776-mm resolution for reconstruction. Evenly spaced angles were subsampled and used for varying the number of projections for the image reconstruction. To assess the performance of our GP-BB algorithm, we have implemented and compared with three compressed sensing-type algorithms, the two of which are popular and published (forward-backward splitting techniques), and the other one with a basic line-search technique. In addition, the conventional Feldkamp-Davis-Kress (FDK) reconstruction of the clinical patient data is compared as well. Results: In comparison with the other compressed sensing-type algorithms, our algorithm showed convergence in {<=}30 iterations whereas other published algorithms need at least 50 iterations in order to reconstruct the Shepp-Logan phantom image. With the CatPhan phantom, the GP-BB algorithm achieved a clinically-reasonable image with 40 projections in 12 iterations, in less than 12.6 s. This is at least an order of magnitude faster in reconstruction time compared with the most recent reports utilizing GPU technology given the same input projections. For the head-and-neck clinical scan, clinically-reasonable images were obtained from 120 projections in 34-78 s converging in 12-30 iterations. In this reconstruction range (i.e., 120 projections) the image quality is visually similar to or better than the conventional FDK reconstructed images using 364 projections. This represents a dose reduction of nearly 67% (120/364 projections) while maintaining a reasonable speed in clinical implementation. Conclusions: In this paper, we proposed a novel, fast, low-dose CBCT reconstruction algorithm using the Barzilai-Borwein step-size calculation. A clinically viable head-and-neck image can be obtained within {approx}34-78 s while simultaneously cutting the dose by approximately 67%. This makes our GP-BB algorithm potentially useful in an on-line image-guided radiation therapy (IGRT).

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2025

Surrogate optimization of variational quantum circuits

Variational quantum eigensolvers are touted as a near-term algorithm capable of impacting many applications. However, the potential has not yet been realized, with few claims of quantum advantage and high resource estimates, especially due to the need for optimization in the presence of noise. Finding algorithms and methods to improve convergence is important to accelerate the capabilities of near-term hardware for VQE or more broad applications of hybrid methods in which optimization is required. To this goal, we look to use modern approaches developed in circuit simulations and stochastic classical optimization, which can be combined to form a surrogate optimization approach to quantum circuits. Using an approximate (classical CPU/GPU) state vector simulator as a surrogate model, we efficiently calculate an approximate Hessian, passed as an input for a quantum processing unit or exact circuit simulator. This method will lend itself well to parallelization across quantum processing units. We demonstrate the capabilities of such an approach with and without sampling noise and a proof-of-principle demonstration on a quantum processing unit utilizing 40 qubits.

Gustafson, Erik J. [RIACS, Mtn. View] (ORCID:00000↗
DOE OSTI2016

Global adjoint tomography: first-generation model

We present the first-generation global tomographic model constructed based on adjoint tomography, an iterative full-waveform inversion technique. Synthetic seismograms were calculated using GPU-accelerated spectral-element simulations of global seismic wave propagation, accommodating effects due to 3-D anelastic crust & mantle structure, topography & bathymetry, the ocean load, ellipticity, rotation, and self-gravitation. Fréchet derivatives were calculated in 3-D anelastic models based on an adjoint-state method. The simulations were performed on the Cray XK7 named ‘Titan’, a computer with 18 688 GPU accelerators housed at Oak Ridge National Laboratory. The transversely isotropic global model is the result of 15 tomographic iterations, which systematically reduced differences between observed and simulated three-component seismograms. Our starting model combined 3-D mantle model S362ANI with 3-D crustal model Crust2.0. We simultaneously inverted for structure in the crust and mantle, thereby eliminating the need for widely used ‘crustal corrections’. We used data from 253 earthquakes in the magnitude range 5.8 ≤ M w ≤ 7.0. We started inversions by combining ~30 s body-wave data with ~60 s surface-wave data. The shortest period of the surface waves was gradually decreased, and in the last three iterations we combined ~17 s body waves with ~45 s surface waves. We started using 180 min long seismograms after the 12th iteration and assimilated minor- and major-arc body and surface waves. The 15th iteration model features enhancements of well-known slabs, an enhanced image of the Samoa/Tahiti plume, as well as various other plumes and hotspots, such as Caroline, Galapagos, Yellowstone and Erebus. Furthermore, we see clear improvements in slab resolution along the Hellenic and Japan Arcs, as well as subduction along the East of Scotia Plate, which does not exist in the starting model. Point-spread function tests demonstrate that we are approaching the resolution of continental-scale studies in some areas, for example, underneath Yellowstone. Here, this is a consequence of our multiscale smoothing strategy in which we define our smoothing operator as a function of the approximate Hessian kernel, thereby smoothing gradients less wherever we have good ray coverage, such as underneath North America.

58 GEOSCIENCES↗
NASA NTRS1982

Minimization versus homotopy algorithms

The relative merits and demerits of the minimization techniques are assessed using globally convergent quasi-Newton algorithms on the one hand and the homotopy algorithms on the other hand for the solution of problems of nonlinear structural analysis. Like the homotopy algorithms, the globally convergent quasi-Newton algorithms are equally suited for the solution of the nonlinear equations of structural analysis directly without having to pose the problem as an equivalent minimization problem. In the close neighborhood of the limit and bifurcation points quasi-Newton algorithms experience difficulties. Homotopy algorithms are robust for practically all types of nonlinear problems but are computationally not as cost effective since they provide an extremely accurate prediction of the response by calculating it as a large number of points. Globally convergent algorithms can perform well with very approximate Hessians, while homotopy algorithms require extremely accurate Hessians. While quasi-Newton algorithms can be very easily structured to exploit sparsity and symmetry, homotopy algorithms are not presently so structured and would require special modifications for exploitation of such features without sacrificing robustness and global convergence.

Kamat, M. P.↗
NASA NTRS1991

A parallel trajectory optimization tool for aerospace plane guidance

A parallel trajectory optimization algorithm is being developed. One possible mission is to provide real-time, on-line guidance for the National Aerospace Plane. The algorithm solves a discrete-time problem via the augmented Lagrangian nonlinear programming algorithm. The algorithm exploits the dynamic programming structure of the problem to achieve parallelism in calculating cost functions, gradients, constraints, Jacobians, Hessian approximations, search directions, and merit functions. Special additions to the augmented Lagrangian algorithm achieve robust convergence, achieve (almost) superlinear local convergence, and deal with constraint curvature efficiency. The algorithm can handle control and state inequality constraints such as angle-of-attack and dynamic pressure constraints. Portions of the algorithm have been tested. The nonlinear programming core algorithm performs well on a variety of static test problems and on an orbit transfer problem. The parallel search direction algorithm can reduce wall clock time by a factor of 10 for this part of the computation task.

Psiaki, Mark L.↗
NASA NTRSDate not supplied

Surrogate Optimization for Quantum Circuits

Variational quantum Eigensolvers are touted as a near-term algorithm capable of impacting many applications. However, the potential has yet to be realized with few claims of quantum advantage and high resource estimates mainly due to the need for optimization in the presence of noise. Finding algorithms and methods to improve the convergence is essential to accelerate the capabilities of near-term hardware for VQE or more broad applications of hybrid methods in which optimization is required. To this goal we look to use modern approaches recently developed in circuit simulations and stochastic classical optimization that can be combined in a surrogate optimization approach to classical circuits. Using an approximate state vector simulator, we efficiently calculate an approximate Hessian, fed as an input for a detailed quantum circuit simulator. We demonstrate the capabilities of such an approach with and without sampling noise. We also show that this method outperforms Powell in the presence of quantum circuit shot noise by a factor of 2-4

quantum computing↗
DOE OSTI2010

EMGeo 3.0

EMGeo 3.0 an algorithm that improves both the computational capabilities of joint 3D electromagnetic, EM and magnetotelluric and acoustic seismic wavefield simulation and inverse modeling. This seismic wavefield module requires that data be transformed first to the Laplace-Fourier Domain before processing. The inverse modeling is based upon non-liner conjugated gradients and 3D finite differences approximations of the seismic, EM and MT wavefields. Improved modeling efficiencies are obtain by separating the modeling grid from the simulation grid. This grid separation method allows for treatment of very large data sets and imaging volumes. Further computational efficiency is obtained by combining different levels of parallezation using the Message Passing Interface (MPI). Bound constraints are employed in the imaging process to insure stability. Additional acceleration of the inverse modeling is achieved by preconditioning the conjugate gradient optimizer using an approximate Hessian. The algorithm includes improved capabilities to accurately treat models that exhibit transverse anisotropy in the electrical conductivity. However, the seismic acoustic velocity is assumed to be isotropic. Models exhibiting topography and bathymetry can be treated. The software also includes a set of pre and post processing tools for designing input model meshes and data plotting.

Gregory Newman, Michael Commer↗