Search NASA⌕ Search

SEARCH · Search NASA

Results for “Gradient descent”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Quantum optimization algorithms: Energetic implications

Since the dawn of quantum computing (QC), theoretical developments like Shor's algorithm proved the conceptual superiority of QC over traditional computing. However, such quantum supremacy claims are difficult to achieve in practice because of the technical challenges of realizing noiseless qubits. In the near future, QC applications will need to rely on noisy quantum devices that offload part of their work to classical devices. One way to achieve this is by using parameterized quantum circuits in optimization or even in machine learning tasks. The energy requirements of quantum algorithms have not yet been studied extensively. Here in this article, we explore several optimization algorithms using both theoretical insights and numerical experiments to understand their impact on energy consumption. Specifically, we highlight why and how algorithms like quantum natural gradient descent, simultaneous perturbation stochastic approximations or circuit learning methods, are at least 2x to 4x more energy efficient than their classical counterparts; why feedback-based quantum optimization is energy-inefficient; and how techniques like Rosalin can improve the energy efficiency of other algorithms by a factor of ≥2 0 x. Finally, we use the NchooseK high-level programming model to run optimization problems on both gate-based quantum computers and quantum annealers. Empirical data indicate that these optimization problems run faster, have better success rates, and consume less energy on quantum annealers than on their gate-based counterparts.

97 MATHEMATICS AND COMPUTING↗

Statistical Shape Modeling of Cam Femoroacetabular Impingement

In this study, statistical shape modeling (SSM) was used to quantify three-dimensional (3D) variation and morphologic differences between femurs with and without cam femoroacetabular impingement (FAI). 3D surfaces were generated from CT scans of femurs from 41 controls and 30 cam FAI patients. SSM correspondence particles were optimally positioned on each surface using a gradient descent energy function. Mean shapes for control and patient groups were defined from the resulting particle configurations. Morphological differences between group mean shapes and between the control mean and individual patients were calculated. Principal component analysis was used to describe anatomical variation present in both groups. The first 6 modes (or principal components) captured statistically significant shape variations, which comprised 84% of cumulative variation among the femurs. Shape variation was greatest in femoral offset, greater trochanter height, and the head-neck junction. The mean cam femur shape protruded above the control mean by a maximum of 3.3 mm with sustained protrusions of 2.5-3.0 mm along the anterolateral head-neck junction and distally along the anterior neck, corresponding well with reported cam lesion locations and soft-tissue damage. This study provides initial evidence that SSM can describe variations in femoral morphology in both controls and cam FAI patients and may be useful for developing new measurements of pathological anatomy. SSM may also be applied to characterize cam FAI severity and provide templates to guide patient-specific surgical resection of bone.

statistical shape modeling↗

Probabilistic partition of unity networks for high–dimensional regression problems

We explore the probabilistic partition of unity network (PPOU-Net) model in the context of high-dimensional regression problems and propose a general framework focusing on adaptive dimensionality reduction. With the proposed framework, the target function is approximated by a mixture of experts model on a low-dimensional manifold, where each cluster is associated with a fixed-degree polynomial. We present a training strategy that leverages the expectation maximization (EM) algorithm. During the training, we alternate between (i) applying gradient descent to update the DNN coefficients; and (ii) using closed-form formulae derived from the EM algorithm to update the mixture of experts model parameters. Under the probabilistic formulation, step (ii) admits the form of embarrassingly paralleliazable weighted least-squares solves. The PPOU-Nets consistently outperform the baseline fully-connected neural networks of comparable sizes in numerical experiments of various data dimensions. Here, we also explore the proposed model in applications of quantum computing, where the PPOU-Nets act as surrogate models for cost landscapes associated with variational quantum circuits.

97 MATHEMATICS AND COMPUTING↗

A rotorcraft in-flight ice detection framework using computational aeroacoustics and Bayesian neural networks

Abstract This work develops a novel ice detection framework specifically suitable for rotorcraft using computational aeroacoustics and Bayesian neural networks. In an offline phase of the work, the acoustic signature of glaze and rime ice shapes on an oscillating wing are computed. In addition, the aerodynamic performance indicators corresponding to the ice shapes are also monitored. These performance indicators include the lift, drag, and moment coefficients. A Bayesian neural network is subsequently trained using projected Stein variational gradient descent to create a mapping from the acoustic signature generated by the iced wings to predict their performance indicators along with quantified uncertainty that is highly important for time- and safety-critical decision-making scenarios. While the training is carried out fully offline, usage of the Bayesian neural network to make predictions can be conducted rapidly online allowing for an ice detection system that can be used in real time and in-flight.

42 ENGINEERING↗

Analysis of a New Variational Model to Restore Point-Like and Curve-Like Singularities in Imaging

The paper is concerned with the analysis of a new variational model to restore point-like and curve-like singularities in biological images. To this aim we investigate the variational properties of a suitable energy which governs these pathologies. Finally in order to realize numerical experiments we minimize, in the discrete setting, a regularized version of this functional by fast descent gradient scheme.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Prediction of gas hydrate saturation using machine learning and optimal set of well-logs

We report resistivity and acoustic logs are widely used to estimate gas hydrate saturation in various sedimentary systems using one of the two popular methods ((1) acoustic velocity and (2) electrical resistivity), but the limitations of these two methods are often overlooked, which include (i) well-specific calibration of empirical exponents in the electrical resistivity method, (ii) assumption of known pore morphology for gas hydrates in the acoustic velocity method, and (iii) presence of unknown mineralogy and bulk modulus terms in the acoustic velocity method. NMR-density porosity-derived gas hydrate saturation based on the analysis of the transverse magnetization relaxation time (T2) is considered the most precise method, but acquisition of NMR-based logs is limited at relatively recent drilled sites; additionally, its use in conventional oil and gas reservoirs is not that common due to higher cost and operational deployment limitations associated with acquiring NMR well-logs. This study proposes a new method that predicts gas hydrate saturation (S h ) for any well using porosity, bulk density, and compressional wave (P wave) velocity well-logs with neural network (or stochastic gradient descent regression) without any well-specific calibration and/or other aforementioned shortcomings of the existing methods. The method is developed by examining the underlying dependency between S h and different combinations of well-logs, chosen from 6 routine logs, with 12 different machine learning (ML) algorithms. The accuracy of the proposed method in predicting S h is ~ 84%, which is better than the accuracy of seismic and electrical resistivity methods (≤ 75%) per the results reported by three different studies. The robustness of the method in the specific case of permafrost-associated gas hydrates is demonstrated with well-log data from two wells drilled on the Alaska North Slope.

58 GEOSCIENCES↗

A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks

To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (Biometrika 81(3):425–455, 1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform simulated and real data Monte Carlo experiments to quantify the effectiveness of our approach.

97 MATHEMATICS AND COMPUTING↗

MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most popular approach, which essentially optimizes for a latent vector that minimizes the discrepancy between a generated image and the given observation. However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount. Such corruptions are common in the real world, for example images in the wild come with unknown crops, rotations, missing pixels, or other kinds of non-linear distributional shifts which break current encoding methods, rendering downstream applications unusable. To address this, we propose corruption mimicking—a new robust projection technique, that utilizes a surrogate network to approximate the unknown corruption directly at test time, without the need for additional supervision or data augmentation. The proposed method is significantly more robust than PGD and other competing methods under a wide variety of corruptions, thereby enabling a more effective use of GANs in real-world applications. Finally, more importantly, we show that our approach produces state-of-the-art performance in several GAN-based applications—anomaly detection, domain adaptation, and adversarial defense, that benefit from an accurate projection.

97 MATHEMATICS AND COMPUTING↗

Learning generative neural networks with physics knowledge

Deep generative neural networks have enabled modeling complex distributions, but incorporating physics knowledge into the neural networks is still challenging and is at the core of current physics-based machine learning research. To this end, we propose a physics generative neural network (PhysGNN), a new class of generative neural networks for learning unknown distributions in a physical system described by partial differential equations (PDE). PhysGNN couples PDE systems with generative neural networks. It is a fully differentiable model that allows back-propagation of gradients through both numerical PDE solvers and generative neural networks, and is trained by minimizing the discrete Wasserstein distance between generated and observed probability distributions of the PDE outputs using the stochastic gradient descent method. Moreover, PhysGNN does not require adversarial training like standard generative neural networks, which offers better stability than adversarial training. We show that PhysGNN can learn complex distributions in stochastic inverse problems, where conventional methods such as maximum likelihood estimation and momentum matching methods may be inapplicable when little knowledge is known about the form of unknown distributions or the physical model is too complex. Furthermore, our method allows physics-based generative neural network training for learning complex distributions in the context of differential equations.

97 MATHEMATICS AND COMPUTING↗

Object-space optimization of tomographic reconstructions for additive manufacturing

Volumetric 3D printing motivated by computed axial lithography enables rapid printing of homogeneous parts but requires a high dimensionality gradient-descent optimization to calculate image sets. Here we introduce a new, simpler approach to image-computation that algebraically optimizes a model of the printed object, significantly improving print accuracy of complex parts under imperfect material and optical precision by improving optical dose contrast between the target and surrounding regions. Quality metrics for volumetric printing are defined and shown to be significantly improved by the new algorithm. The approach is extended beyond binary printing to grayscale control of conversion to enable functionally graded materials. The flexibility of the technique is digitally demonstrated with realistic projector point spread functions, printing around occluding structures, printing with restricted angular range, and incorporation of materials chemistry such as inhibition. Finally, simulations show that the method facilitates new printing modalities such as printing into flat, rather than cylindrical packages to extend the applications of volumetric printing.

36 MATERIALS SCIENCE↗

Application of unsupervised deep learning to image segmentation and in-situ contact angle measurements in a CO 2 -water-rock system

Rock surface wettability is a critical property that regulates multiphase flows in porous media, which can be quantified using the surface contact angle (CA). X-ray micro-computed tomography (μCT) provides an effective approach to in-situ measurements of surface CAs. However, the CA measurement accuracy depends significantly on the quality of CT image segmentation, which is the clustering of CT pixels into separate phases. Inspired by this, we developed a deep learning (DL)-based CA measurement workflow. Motivated by the recent tremendous progress in unsupervised learning techniques and aiming to avoid expensive manual data annotations, an unsupervised DL pipeline for CT image segmentation was proposed and implemented, which includes unsupervised model training and post-processing. The unsupervised model training was driven by a novel loss function constrained with feature similarity and spatial continuity and implemented by iterative forward and backward paths; the former clustered the pixel-wise feature vectors extracted by convolution neural networks, whereas the latter updated the parameters using gradient descent. An over-segmentation strategy was adopted for model training. The post-processing steps based on agglomerative hierarchical clustering (AHC) were implemented to further merge the over-segmented model output to the desired cluster number, which is intended to improve the efficiency of image segmentation. The developed unsupervised DL pipeline was compared with other commonly-used image segmentation methods using pixel-wise and physics-based evaluation metrics on a synthetic raw-image dataset, which had a known ground truth. The unsupervised DL pipeline showed the best performance. Next, the segmented images were input to an automatic CA measurement tool, and the results were validated by comparisons with manual measurements. The CA values from the manual and automatic measurements showed similar distributions and statistical properties. The automatic measurement demonstrated a wider spectrum because of the much larger number of measurement data points. The primary novelty of the unsupervised DL pipeline developed in this study lies in the novel loss function and the over-segmentation strategy associated with AHC post-processing. Finally, the workflow has been proven an efficient tool for pore-scale wettability characterization, which has a wide range of applications in fundamental studies of multiphase flows in natural porous media, which have critical implications to geological carbon sequestration, hydrocarbon energy recovery, and contaminant transport in groundwater.

42 ENGINEERING↗

Stochastic Optimization and Uncertainty Quantification of Natrium-based Nuclear-Renewable Energy Systems for Flexible Power Applications in Deregulated Markets

Rapid integration of variable renewable energy sources (VRES) has made modeling and stochastic optimization of hybrid energy systems crucial for studying their long-term performance and viability. However, most studies have focused on just historical data, which may be unreliable for capturing short-term fluctuations, rare events, and long-term patterns of energy demand, price, and the variability of renewable energy sources. For this study, optimal synthetic time series models were developed using Wasserstein distance. The models were validated by comparing the key statistical measures against those of the historical data. They were then used to optimize the integrated Natrium-style advanced energy systems and their long-term (30 years) economics. The stochastic model performs bi-level optimization to find the optimal sizes for the balance of plant and thermal energy storage, while also optimizing energy dispatch to achieve the maximum net present value. In studies of two deregulated markets (California ISO and the Electric Reliability Council of Texas), the integrated Natrium-style system performed better in CAISO than in ERCOT, given higher and more consistent electricity prices during peak-demand periods. The potentially enlarged cost associated with the variable operation and maintenance of the TES system also plays a significant role in driving the system sizing, thus its impacts on the system are investigated in detail through comparison against a baseline case. The study also finds that the bi-level optimization results based on stochastic gradient descent closely match the grid search results. The uncertainty quantification of the stochastic signals provides further NPV-related insights and probability distributions for the case studies. The normal standard error of the mean of NPV for the case with and without TES VOM for CAISO were found to be 7.73M (plus-minus sign) 1.09M USD and 104.99M (plus-minus sign) 1.25M USD, respectively based on a 95% confidence. Given the relatively small NPV variance based on 150 samples, the analysis affords the most robust possible prediction of the techno-economic performance of the integrated Natrium-style energy systems.

25 ENERGY STORAGE↗

An adaptive sampling augmented Lagrangian method for stochastic optimization with deterministic constraints

The primary goal of this paper is to provide an efficient solution algorithm based on the augmented Lagrangian framework for optimization problems with a stochastic objective function and deterministic constraints. Our main contribution is combining the augmented Lagrangian framework with adaptive sampling, resulting in an efficient optimization methodology validated with practical examples. To achieve the presented efficiency, here we consider inexact solutions for the augmented Lagrangian subproblems, and through an adaptive sampling mechanism, we control the variance in the gradient estimates. Furthermore, we analyze the theoretical performance of the proposed scheme by showing equivalence to a gradient descent algorithm on a Moreau envelope function, and we prove sublinear convergence for convex objectives and linear convergence for strongly convex objectives with affine equality constraints. The worst-case sample complexity of the resulting algorithm, for an arbitrary choice of penalty parameter in the augmented Lagrangian function, is $\mathscr{O}$(ϵ -3-δ ) , where ϵ > 0 is the expected error of the solution and δ > 0 is a user-defined parameter. If the penalty parameter is chosen to be $\mathscr{O}$(ϵ -1 ), we demonstrate that the result can be improved to $\mathscr{O}$(ϵ -2 ) , which is competitive with the other methods employed in the literature. Moreover, if the objective function is strongly convex with affine equality constraints, we obtain $\mathscr{O}$(ϵ -1 log(1/ϵ)) complexity. Finally, we empirically verify the performance of our adaptive sampling augmented Lagrangian framework in machine learning optimization and engineering design problems, including topology optimization of a heat sink with environmental uncertainty.

97 MATHEMATICS AND COMPUTING↗

Bayesian inference of heterogeneous epidemic models: Application to COVID-19 spread accounting for long-term care facilities

Here we propose a high dimensional Bayesian inference framework for learning heterogeneous dynamics of a COVID-19 model, with a specific application to the dynamics and severity of COVID-19 inside and outside long-term care (LTC) facilities. We develop a heterogeneous compartmental model that accounts for the heterogeneity of the time-varying spread and severity of COVID-19 inside and outside LTC facilities, which is characterized by time-dependent stochastic processes and time-independent parameters in ~ 1500 dimensions after discretization. To infer these parameters, we use reported data on the number of confirmed, hospitalized, and deceased cases with suitable post-processing in both a deterministic inversion approach with appropriate regularization as a first step, followed by Bayesian inversion with proper prior distributions. To address the curse of dimensionality and the ill-posedness of the high-dimensional inference problem, we propose use of a dimension-independent projected Stein variational gradient descent method, and demonstrate the intrinsic low-dimensionality of the inverse problem. We present inference results with quantified uncertainties for both New Jersey and Texas, which experienced different epidemic phases and patterns. Moreover, we also present forecasting and validation results based on the empirical posterior samples of our inference for the future trajectory of COVID-19.

42 ENGINEERING↗

D2NO: Efficient handling of heterogeneous input function spaces with distributed deep neural operators

Neural operators have been applied in various scientific fields, such as solving parametric partial differential equations, dynamical systems with control, and inverse problems. However, challenges arise when dealing with input functions that exhibit heterogeneous properties, requiring multiple sensors to handle functions with minimal regularity. To address this issue, discretization-invariant neural operators have been used, allowing the sampling of diverse input functions with different sensor locations. However, existing frameworks still require an equal number of sensors for all functions. We propose a novel distributed approach to further relax the discretization requirements and solve the heterogeneous dataset challenges. Our method involves partitioning the input function space and processing individual input functions using independent and separate neural networks. A centralized neural network is used to handle shared information across all output functions. This distributed methodology reduces the number of gradient descent back-propagation steps, improving efficiency while maintaining accuracy. Here, we demonstrate that the corresponding neural network is a universal approximator of continuous nonlinear operators and present three numerical examples to validate its performance.

97 MATHEMATICS AND COMPUTING↗

Volumetric Modulated Arc Therapy: Planning and Evaluation for Prostate Cancer Cases

Purpose: To develop an optimization method using volumetric modulated arc therapy (VMAT) and evaluate VMAT plans relative to the standard intensity-modulated radiotherapy (IMRT) approach in prostate cancer. Methods and Materials: A single gantry rotation was modeled using 177 equispaced beams. Multileaf collimator apertures and dose rates were optimized with respect to gantry angle subject to dose-volume-based objectives. Our VMAT implementation used conjugate gradient descent to optimize dose rate, and stochastic sampling to find optimal multileaf collimator leaf positions. A treatment planning study of 11 prostate cancer patients with a prescription dose of 86.4 Gy was performed to compare VMAT with a standard five-field IMRT approach. Plan evaluation statistics included the percentage of planning target volume (PTV) receiving 95% of prescribed dose (V95), dose to 95% of PTV (D95), mean PTV dose, tumor control probability, and dosimetric endpoints of normal organs, whereas monitor unit (MU) and delivery time were used to assess delivery efficiency. Results: Patient-averaged PTV V95, D95, mean dose, and tumor control probability in VMAT plans were 96%, 82.6 Gy, 88.5 Gy, and 0.920, respectively, vs. 97%, 84.0 Gy, 88.9 Gy, and 0.929 in IMRT plans. All critical structure dose requirements were met. The VMAT plans presented better rectal wall sparing, with a reduction of 1.5% in normal tissue complication probability. An advantage of VMAT plans was that the average number of MUs (290 MU) was less than for IMRT plans (642 MU). Conclusion: The VMAT technique can reduce beam on time by up to 55% while maintaining dosimetric quality comparable to that of the standard IMRT approach.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima.

machine leaning, Bad minima, Inverse problems, Phy↗