SEARCH · Search NASA

Results for “Hessian approximation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

14 records

All sources NASA NTRS DOE OSTI

NASA NTRS1998

Accelerated Training for Large Feedforward Neural Networks

In this paper we introduce a new training algorithm, the scaled variable metric (SVM) method. Our approach attempts to increase the convergence rate of the modified variable metric method. It is also combined with the RBackprop algorithm, which computes the product of the matrix of second derivatives (Hessian) with an arbitrary vector. The RBackprop method allows us to avoid computationally expensive, direct line searches. In addition, it can be utilized in the new, 'predictive' updating technique of the inverse Hessian approximation. We have used directional slope testing to adjust the step size and found that this strategy works exceptionally well in conjunction with the Rbackprop algorithm. Some supplementary, but nevertheless important enhancements to the basic training scheme such as improved setting of a scaling factor for the variable metric update and computationally more efficient procedure for updating the inverse Hessian approximation are presented as well. We summarize by comparing the SVM method with four first- and second- order optimization algorithms including a very effective implementation of the Levenberg-Marquardt method. Our tests indicate promising computational speed gains of the new training technique, particularly for large feedforward networks, i.e., for problems where the training process may be the most laborious.

Stepniewski, Slawomir W.↗

NASA NTRS1987

Recursive inverse kinematics for robot arms via Kalman filtering and Bryson-Frazier smoothing

This paper applies linear filtering and smoothing theory to solve recursively the inverse kinematics problem for serial multilink manipulators. This problem is to find a set of joint angles that achieve a prescribed tip position and/or orientation. A widely applicable numerical search solution is presented. The approach finds the minimum of a generalized distance between the desired and the actual manipulator tip position and/or orientation. Both a first-order steepest-descent gradient search and a second-order Newton-Raphson search are developed. The optimal relaxation factor required for the steepest descent method is computed recursively using an outward/inward procedure similar to those used typically for recursive inverse dynamics calculations. The second-order search requires evaluation of a gradient and an approximate Hessian. A Gauss-Markov approach is used to approximate the Hessian matrix in terms of products of first-order derivatives. This matrix is inverted recursively using a two-stage process of inward Kalman filtering followed by outward smoothing. This two-stage process is analogous to that recently developed by the author to solve by means of spatial filtering and smoothing the forward dynamics problem for serial manipulators.

Rodriguez, G.↗

NASA NTRS1982

Minimization versus homotopy algorithms

The relative merits and demerits of the minimization techniques are assessed using globally convergent quasi-Newton algorithms on the one hand and the homotopy algorithms on the other hand for the solution of problems of nonlinear structural analysis. Like the homotopy algorithms, the globally convergent quasi-Newton algorithms are equally suited for the solution of the nonlinear equations of structural analysis directly without having to pose the problem as an equivalent minimization problem. In the close neighborhood of the limit and bifurcation points quasi-Newton algorithms experience difficulties. Homotopy algorithms are robust for practically all types of nonlinear problems but are computationally not as cost effective since they provide an extremely accurate prediction of the response by calculating it as a large number of points. Globally convergent algorithms can perform well with very approximate Hessians, while homotopy algorithms require extremely accurate Hessians. While quasi-Newton algorithms can be very easily structured to exploit sparsity and symmetry, homotopy algorithms are not presently so structured and would require special modifications for exploitation of such features without sacrificing robustness and global convergence.

Kamat, M. P.↗

NASA NTRS1991

A parallel trajectory optimization tool for aerospace plane guidance

A parallel trajectory optimization algorithm is being developed. One possible mission is to provide real-time, on-line guidance for the National Aerospace Plane. The algorithm solves a discrete-time problem via the augmented Lagrangian nonlinear programming algorithm. The algorithm exploits the dynamic programming structure of the problem to achieve parallelism in calculating cost functions, gradients, constraints, Jacobians, Hessian approximations, search directions, and merit functions. Special additions to the augmented Lagrangian algorithm achieve robust convergence, achieve (almost) superlinear local convergence, and deal with constraint curvature efficiency. The algorithm can handle control and state inequality constraints such as angle-of-attack and dynamic pressure constraints. Portions of the algorithm have been tested. The nonlinear programming core algorithm performs well on a variety of static test problems and on an orbit transfer problem. The parallel search direction algorithm can reduce wall clock time by a factor of 10 for this part of the computation task.

Psiaki, Mark L.↗

NASA NTRSDate not supplied

Surrogate Optimization for Quantum Circuits

Variational quantum Eigensolvers are touted as a near-term algorithm capable of impacting many applications. However, the potential has yet to be realized with few claims of quantum advantage and high resource estimates mainly due to the need for optimization in the presence of noise. Finding algorithms and methods to improve the convergence is essential to accelerate the capabilities of near-term hardware for VQE or more broad applications of hybrid methods in which optimization is required. To this goal we look to use modern approaches recently developed in circuit simulations and stochastic classical optimization that can be combined in a surrogate optimization approach to classical circuits. Using an approximate state vector simulator, we efficiently calculate an approximate Hessian, fed as an input for a detailed quantum circuit simulator. We demonstrate the capabilities of such an approach with and without sampling noise. We also show that this method outperforms Powell in the presence of quantum circuit shot noise by a factor of 2-4

quantum computing↗

NASA NTRS1985

A Residuals Approach to Filtering, Smoothing and Identification for Static Distributed Systems

An approach for state estimation and identification of spatially distributed parameters embedded in static distributed (elliptic) system models is advanced. The method of maximum likelihood is used to find parameter values that maximize a likelihood functional for the system model, or equivalently, that minimize the negative logarithm of this functional. To find the minimum, a Newton-Raphson search is conducted that from an initial estimate generates a convergent sequence of parameter estimates. For simplicity, a Gauss-Markov approach is used to approximate the Hessian in terms of products of first derivatives. The gradient and approximate Hessian are computed by first arranging the negative log likelihood functional into a form based on the square root factorization of the predicted covariance of the measurement process. The resulting data processing approach, referred to here by the new term of predicted data covariance square root filtering, makes the gradient and approximate Hessian calculations very simple. A closely related set of state estimates is also produced by the maximum likelihood method: smoothed estimates that are optimal in a conditional mean sense and filtered estimates that emerge from the predicted data covariance square root filter.

Rodriguez, G.↗

NASA NTRS1996

Analysis of the Hessian for Aerodynamic Optimization: Inviscid Flow

In this paper we analyze inviscid aerodynamic shape optimization problems governed by the full potential and the Euler equations in two and three dimensions. The analysis indicates that minimization of pressure dependent cost functions results in Hessians whose eigenvalue distributions are identical for the full potential and the Euler equations. However the optimization problems in two and three dimensions are inherently different. While the two dimensional optimization problems are well-posed the three dimensional ones are ill-posed. Oscillations in the shape up to the smallest scale allowed by the design space can develop in the direction perpendicular to the flow, implying that a regularization is required. A natural choice of such a regularization is derived. The analysis also gives an estimate of the Hessian's condition number which implies that the problems at hand are ill-conditioned. Infinite dimensional approximations for the Hessians are constructed and preconditioners for gradient based methods are derived from these approximate Hessians.

Arian, Eyal↗

NASA NTRS2015

A Versatile Nonlinear Method for Predictive Modeling

As computational fluid dynamics techniques and tools become widely accepted for realworld practice today, it is intriguing to ask: what areas can it be utilized to its potential in the future. Some promising areas include design optimization and exploration of fluid dynamics phenomena (the concept of numerical wind tunnel), in which both have the common feature where some parameters are varied repeatedly and the computation can be costly. We are especially interested in the need for an accurate and efficient approach for handling these applications: (1) capturing complex nonlinear dynamics inherent in a system under consideration and (2) versatility (robustness) to encompass a range of parametric variations. In our previous paper, we proposed to use first-order Taylor expansion collected at numerous sampling points along a trajectory and assembled together via nonlinear weighting functions. The validity and performance of this approach was demonstrated for a number of problems with a vastly different input functions. In this study, we are especially interested in enhancing the method's accuracy; we extend it to include the second-orer Taylor expansion, which however requires a complicated evaluation of Hessian matrices for a system of equations, like in fluid dynamics. We propose a method to avoid these Hessian matrices, while maintaining the accuracy. Results based on the method are presented to confirm its validity.

nonlinear modeling↗

NASA NTRS1979

A cubic extended interior penalty function for structural optimization

This paper describes an optimization procedure for the minimum weight design of complex structures. The procedure is based on a new cubic extended interior penalty function (CEIPF) used with the sequence of unconstrained minimization technique (SUMT) and Newton's method. The Hessian matrix of the penalty function is approximated using only constraints and their derivatives. The CEIPF is designed to minimize the error in the approximation of the Hessian matrix, and as a result the number of structural analyses required is small and independent of the number of design variables. Three example problems are reported. The number of structural analyses is reduced by as much as 50 per cent below previously reported results.

Prasad, B.↗

NASA NTRS1998

A Preconditioning Method for Shape Optimization Governed by the Euler Equations

We consider a classical aerodynamic shape optimization problem subject to the compressible Euler flow equations. The gradient of the cost functional with respect to the shape variables is derived with the adjoint method at the continuous level. The Hessian (second order derivative of the cost functional with respect to the shape variables) is approximated also at the continuous level, as first introduced by Arian and Ta'asan (1996). The approximation of the Hessian is used to approximate the Newton step which is essential to accelerate the numerical solution of the optimization problem. The design space is discretized in the maximum dimension, i.e., the location of each point on the intersection of the computational mesh with the airfoil is taken to be an independent design variable. We give numerical examples for 86 design variables in two different flow speeds and achieve an order of magnitude reduction in the cost functional at a computational effort of a full solution of the analysis partial differential equation (PDE).

Arian, Eyal↗

NASA NTRS1990

Maximum likelihood tuning of a vehicle motion filter

This paper describes the use of maximum likelihood parameter estimation unknown parameters appearing in a nonlinear vehicle motion filter. The filter uses the kinematic equations of motion of a rigid body in motion over a spherical earth. The nine states of the filter represent vehicle velocity, attitude, and position. The inputs to the filter are three components of translational acceleration and three components of angular rate. Measurements used to update states include air data, altitude, position, and attitude. Expressions are derived for the elements of filter matrices needed to use air data in a body-fixed frame with filter states expressed in a geographic frame. An expression for the likelihood functions of the data is given, along with accurate approximations for the function's gradient and Hessian with respect to unknown parameters. These are used by a numerical quasi-Newton algorithm for maximizing the likelihood function of the data in order to estimate the unknown parameters. The parameter estimation algorithm is useful for processing data from aircraft flight tests or for tuning inertial navigation systems.

Trankle, Thomas L.↗

NASA NTRS1997

Convergence Estimates for Multidisciplinary Analysis and Optimization

A quantitative analysis of coupling between systems of equations is introduced. This analysis is then applied to problems in multidisciplinary analysis, sensitivity, and optimization. For the sensitivity and optimization problems both multidisciplinary and single discipline feasibility schemes are considered. In all these cases a "convergence factor" is estimated in terms of the Jacobians and Hessians of the system, thus it can also be approximated by existing disciplinary analysis and optimization codes. The convergence factor is identified with the measure for the "coupling" between the disciplines in the system. Applications to algorithm development are discussed. Demonstration of the convergence estimates and numerical results are given for a system composed of two non-linear algebraic equations, and for a system composed of two PDEs modeling aeroelasticity.

Arian, Eyal↗

NASA NTRS1988

Advances in dual algorithms and convex approximation methods

A new algorithm for solving the duals of separable convex optimization problems is presented. The algorithm is based on an active set strategy in conjunction with a variable metric method. This first order algorithm is more reliable than Newton's method used in DUAL-2 because it does not break down when the Hessian matrix becomes singular or nearly singular. A perturbation technique is introduced in order to remove the nondifferentiability of the dual function which arises when linear constraints are present in the approximate problem.

Smaoui, H.↗

NASA NTRS1994

A Numerical Comparison of Barrier and Modified Barrier Methods for Large-Scale Bound-Constrained Optimization

When a classical barrier method is applied to the solution of a nonlinear programming problem with inequality constraints, the Hessian matrix of the barrier function becomes increasingly ill-conditioned as the solution is approached. As a result, it may be desirable to consider alternative numerical algorithms. We compare the performance of two methods motivated by barrier functions. The first is a stabilized form of the classical barrier method, where a numerically stable approximation to the Newton direction is used when the barrier parameter is small. The second is a modified barrier method where a barrier function is applied to a shifted form of the problem, and the resulting barrier terms are scaled by estimates of the optimal Lagrange multipliers. The condition number of the Hessian matrix of the resulting modified barrier function remains bounded as the solution to the constrained optimization problem is approached. Both of these techniques can be used in the context of a truncated-Newton method, and hence can be applied to large problems, as well as on parallel computers. In this paper, both techniques are applied to problems with bound constraints and we compare their practical behavior.

Nash, Stephen G.↗