Search NASA⌕ Search

SEARCH · Search NASA

Results for “Loss function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima.

machine leaning, Bad minima, Inverse problems, Phy↗

Efficient Training of Deep Neural Operator Networks via Randomized Sampling

Neural operators (NOs) employ deep neural networks to learn the mappings between infinitedimensional function spaces. Deep operator network (DeepONet), a popular NO architecture, has demonstrated success in the real-time prediction of complex dynamics across various scientific and engineering applications. In this work, we introduce a random sampling technique to be adopted during the training of DeepONet, aimed at improving the generalization ability of the model, while significantly reducing the computational time. The proposed approach targets the trunk network of the DeepONet model that outputs the basis functions corresponding to the spatiotemporal locations of the bounded domain on which the physical system is defined. While constructing the loss function, DeepONet training traditionally considers a uniform grid of spatiotemporal points at which all the output functions are evaluated for each iteration. This approach leads to a larger batch size, resulting in poor generalization and increased memory demands, due to the limitations of the stochastic gradient descent (SGD) optimizer. The proposed random sampling over the inputs of the trunk net mitigates these challenges, improving generalization and reducing the memory requirements during training, resulting in significant computational gains. We validate our hypothesis through three benchmark examples, demonstrating substantial reductions in training time while achieving comparable or lower overall test errors relative to the traditional training approach. Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet, offering a promising avenue for improving the framework’s performance in modeling complex physical systems.

Karumuri, Sharmila [Department of Civil & Systems ↗

Bayesian Model Calibration for Extrapolative Prediction via Gibbs Posteriors

The current standard Bayesian approach to model calibration, which assigns a Gaussian process prior to the discrepancy term, often suffers from issues of unidentifiability and computational complexity and instability. When the goal is to quantify uncertainty in physical parameters for extrapolative prediction, then there is no need to perform inference on the discrepancy term. With this in mind, we introduce Gibbs posteriors as an alternative Bayesian method for model calibration, which updates the prior with a loss function connecting the data to the parameter. The target of inference is the physical parameter value which minimizes the expected loss. We propose to tune the loss scale of the Gibbs posterior to maintain nominal frequentist coverage under assumptions of the form of model discrepancy, and present a bootstrap implementation for approximating coverage rates. Our approach is highly modular, allowing an analyst to easily encode a wide variety of such assumptions. Furthermore, we provide a principled method of combining posteriors calculated from data subsets. We apply our methods to data from an experiment measuring the material properties of tantalum.

36 MATERIALS SCIENCE↗

Loss of Drp1 function alters OPA1 processing and changes mitochondrial membrane organization

RNAi mediated loss of Drp1 function changes mitochondrial morphology in cultured HeLa and HUVEC cells by shifting the balance of mitochondrial fission and fusion towards unopposed fusion. Over time, inhibition of Drp1 expression results in the formation of a highly branched mitochondrial network along with 'bulge'-like structures. These changes in mitochondrial morphology are accompanied by a reduction in levels of Mitofusin 1 (Mfn1) and 2 (Mfn2) and a modified proteolytic processing of OPA1 isoforms, resulting in the inhibition of cell proliferation. In addition, our data imply that bulge formation is driven by Mfn1 action along with particular proteolytic short-OPA1 (s-OPA1) variants: Loss of Mfn2 in the absence of Drp1 results in an increase of Mfn1 levels along with processed s-OPA1-isoforms, thereby enhancing continuous 'fusion' and bulge formation. Moreover, bulge formation might reflect s-OPA1 mitochondrial membrane remodeling activity, resulting in the compartmentalization of cytochrome c deposits. The proteins Yme1L and PHB2 appeared not associated with the observed enhanced OPA1 proteolysis upon RNAi of Drp1, suggesting the existence of other OPA1 processing controlling proteins. Taken together, Drp1 appears to affect the activity of the mitochondrial fusion machinery by unbalancing the protein levels of mitofusins and OPA1.

60 APPLIED LIFE SCIENCES↗

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. Here, we demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Taken together, our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

97 MATHEMATICS AND COMPUTING↗

Plateau Phenomenon in Gradient Descent Training of RELU Networks: Explanation, Quantification, and Avoidance

The ability of neural networks to provide ‘best in class’ approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods,a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon.analysis is carried out in the setting of univariate ReLU networks. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. Here, the main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.

97 MATHEMATICS AND COMPUTING↗

A Look at the Truths and Misconceptions of the Variational Quantum Eigensolver and the Implications of Overparameterization

In this work, we investigate loss landscapes of the variational quantum eigensolver (VQE) by quantifying the number of local minima through empirical analyses. We focus on minimal models in chemistry and physics so that we can do a complete analysis using more computationally expensive tools. We employ Hessian eigenvalue calculations and the nudged elastic band algorithm to characterize these landscapes. Our results expand upon the existing literature by highlighting the optimization challenges faced by VQE. We find that, as the number of parameters in our ansatz increases, the number of basins increases while the corresponding loss function values converge toward the global minimum value. This observation implies that overparameterization may lead to an ``effective convexity'' in VQE loss landscapes, a phenomenon supported by theoretical and numerical work in classical machine learning.

quantum computing↗

Development and Provision of Functional Foods to Promote Health on Long-Duration Space Missions

During long-duration NASA space missions, such as proposed missions to Mars, astronauts may experience negative physiological effects such as bone loss. Functional foods such as high-lycopene, high-flavonoids and high-omega-3 products and fruits and vegetables may mitigate the negative effects of spaceflight on physiological factors including the bone health of crewmembers. Previous studies showed that current ISS provisions provide high-lycopene and high-omega-3 food items but the variety is limited, which could promote menu fatigue. Bioactive compounds can degrade like other chemical compounds and lose functionality. The native concentrations and stability of bioactive compounds have never been determined in spaceflight foods, and adequate information is not available for commercial products for the storage durations required for space exploration (5 years). The purpose of this task is to develop new spaceflight foods that are high in omega-3 fatty acids, lycopene, or flavonoids, identify commercial products with these bioactive compounds that meet spaceflight requirements, and define the stability of these nutrients in storage to enable purposeful functional food incorporation into the space food system. The impact of storage temperature on the stability of lutein, lycopene, beta-carotene, omega-3 fatty acids, phenolics, anthocyanins and sterols is being studied in 12 ISS menu items stored at three different temperatures (4, 21, 35 degree C) over 2 years. Additionally, nutrient and quality stability are being assessed on a larger food set stored at 21 degree C over 2 years that contains twelve newly developed foods, 10 commercial products repackaged to spaceflight requirements, and another 5 current ISS menu items expected to be good sources of omega-3 fatty acids, lycopene, or flavonoids. All items were shipped overnight to the Linus Pauling Institute at Oregon State University (Corvalis, OR) after processing and 1-year of storage and analyzed for bioactive compound concentrations. Sensory evaluation was conducted on the newly developed functional foods and commercial products with untrained panelists (n is greater than or equal to 25) using a 9-point Hedonic scale to test sensory attributes and overall acceptability after processing and 1-year of storage (21 degree C). Repeat nutritional and sensory analyses will be conducted in the same foods after the 2-year storage period is completed. The stability of bioactive compounds in the selected foods was dependent on storage temperature and food matrix. Omega-3 showed excellent stability in the analyzed products after 1-year of storage, regardless of the storage temperature; phenolic compounds also showed good stability. Lycopene was more stable in oil-based products rather than water-based products because of the protection that lipids offer to lycopene molecules. Also, lycopene was more stable in freeze-dried products than in high moisture foods. The 12 newly developed functional foods showed good overall acceptability in sensory attributes after processing (average score 7.2 out of 9.0) and maintained sensory quality through 1-year (21 degree C); the overall acceptability was on average 7.1 after storage. Similar behavior was observed for the 10 commercial products after 1 year. The developed products are good sources of omega-3 (both plant and marine), vegetables (7 vegetable-based products), and good sources of carotenoids, such as the Curry Pumpkin Soup and the Sweet and Savory Kale. Nine of the new products, such as Mango Salad, Pickled Beets, and Braised Red Cabbage, are rich in phenolic compounds. Stability of most of the studied nutrients seems to be adequate after 1-year of storage in most of the tested foods. However, storage temperature of the food must be considered during long-duration space missions to achieve stability of all nutrients. Likewise, more information is needed regarding nutrient retention after 2-years of storage to identify nutritional gaps that may be expected over the 5-year shelf life required for a Mars mission. New developed products will be filling a gap in the current space food system to minimize menu fatigue, provide specific nutrients to reduce the negative effects of long-duration space missions and maintain crew members' health. Information about bioactive compounds in developed products after 1-year and 2-year of storage will provide the knowledge base for further product development.

Bermudez-Aguirre, D.↗

FunDiff: diffusion models over function spaces for physics-informed generative modeling

Recent advances in generative modeling-particularly diffusion models and flow matching-have been widely used for synthesizing discrete data such as images and videos. However, adapting these models to physical applications remains challenging, as the quantities of interest are continuous functions governed by complex physical laws. To address this, we introduce FunDiff, an efficient and robust framework for generative modeling in function spaces. FunDiff combines a latent diffusion process with a function autoencoder architecture to handle input functions with varying discretizations, generates continuous functions that can be evaluated at arbitrary locations, and seamlessly incorporate physical priors. These priors are enforced through architectural constraints or physics-informed loss functions, ensuring that generated samples satisfy fundamental physical laws. We theoretically establish minimax optimality guarantees for density estimation in function spaces, demonstrating that diffusion-based estimators achieve optimal convergence rates under suitable regularity conditions. We further demonstrate the practical effectiveness of FunDiff across diverse applications in fluid dynamics and solid mechanics. Empirical results indicate that our method can generate physically consistent samples with high fidelity to the target distribution, and exhibit robustness to noisy and low-resolution data.

Wang, Sifan [Yale University, New Haven, CT (Unite↗

Temperature and flow velocity of the interplanetary gases along solar radii

The velocity distributions along solar radii for hydrogen and helium in interplanetary space are calculated by using the Danby-Camm formula modified with a loss function. From these distributions the radial temperature and radial flow velocity of the interplanetary gases are determined. The effects of solar gravitation and ionization loss, due to charge exchange and photoionization, on the gas temperature and velocity are described.

Wu, F. M.↗

Physics-informed Karhunen-Loeve and Neural Network Approximations for Solving Inverse Differential Equation Problems

Here we present the PI-CKL-NN method for parameter estimation in differential equation (DE) models given sparse measurements of the parameters and states. In the proposed approach, the space- or time-dependent parameters are approximated by Karhunen-Loeve (KL) expansions that are conditioned on the parameters’ measurements, and the states are approximated by deep neural networks (DNNs). The unknown weights in the KL expansions and DNNs are found my minimizing the cost function that enforces the measurements of the states the DE constraint. Regularization is achieved by adding the l2 norm of the conditional KL coefficients into the loss function. Our approach assumes that the parameter fields are correlated in space or time and enforces the statistical knowledge (the mean and the covariance function) in addition to the DE constraints and measurements as opposed to the physics-informed neural network (PINN) and other similar physics-informed machine learning methods where only DE constraints and data are used for parameter estimation. We use the PI-CKL-NN method for parameter estimation in an ordinary differential equation with an unknown time-dependent parameter and the one- and two-dimensional partial differential diffusion equations with unknown space-dependent diffusion coefficients. We also demonstrate that PI-CKL-NN is more accurate than the PINN method, especially when the observations of the parameters are very sparse

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

New machine learning techniques for simulation-based inference: InferoStatic nets, kernel score estimation, and kernel likelihood ratio estimation

We propose an intuitive, machine-learning approach to multiparameter inference, dubbed the InferoStatic Networks (ISN) method, to model the score and likelihood ratio estimators in cases when the probability density can be sampled but not computed directly. The ISN uses a backend neural network that models a scalar function called the inferostatic potential \varphi φ . In addition, we introduce new strategies, respectively called Kernel Score Estimation (KSE) and Kernel Likelihood Ratio Estimation (KLRE), to learn the score and the likelihood ratio functions from simulated data. We illustrate the new techniques with some toy examples and compare to existing approaches in the literature. We mention en passant some new loss functions that optimally incorporate latent information from simulations into the training procedure.

Kong, Kyoungchul↗

Codebase release 0.1 for infstat

We propose an intuitive, machine-learning approach to multiparameter inference, dubbed the InferoStatic Networks (ISN) method, to model the score and likelihood ratio estimators in cases when the probability density can be sampled but not computed directly. The ISN uses a backend neural network that models a scalar function called the inferostatic potential \varphi φ . In addition, we introduce new strategies, respectively called Kernel Score Estimation (KSE) and Kernel Likelihood Ratio Estimation (KLRE), to learn the score and the likelihood ratio functions from simulated data. We illustrate the new techniques with some toy examples and compare to existing approaches in the literature. We mention en passant some new loss functions that optimally incorporate latent information from simulations into the training procedure.

Kong, Kyoungchul↗

New Machine Learning Techniques for Simulation-Based Inference: InferoStatic Nets, Kernel Score Estimation, and Kernel Likelihood Ratio Estimation

We propose an intuitive, machine-learning approach to multiparameter inference, dubbed the InferoStatic Networks (ISN) method, to model the score and likelihood ratio estimators in cases when the probability density can be sampled but not computed directly. The ISN uses a backend neural network that models a scalar function called the inferostatic potential $\varphi$. In addition, we introduce new strategies, respectively called Kernel Score Estimation (KSE) and Kernel Likelihood Ratio Estimation (KLRE), to learn the score and the likelihood ratio functions from simulated data. We illustrate the new techniques with some toy examples and compare to existing approaches in the literature. We mention en passant some new loss functions that optimally incorporate latent information from simulations into the training procedure.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Transforming the $v$ World: A New Multivariate Transformer Energy Estimator for NOvA

The NOvA Transformer Energy Estimator (Transformer_EE) is a universal machine learning tool currently used to infer the incoming beam neutrino energy and the outgoing lepton energy in both near andfar detectors. It uses a unique, highly flexible framework for simultaneous multivariate prediction that supports many possible loss functions. A spectral reweighting and flattening scheme lessens training bias. A feature noising subroutine enables adversarial-like training, mitigating sensitivities to certain systematic effects at marginal resolution loss at inference time. The state of the Transformer_EE will be reviewed, and its robustness with respect to several NOvA Near and Far Detector systematics highlighted.

Tong, Leon [Minnesota U.] (ORCID:0000000231625965)↗

Triangle Method for Dense ReLU Layers [SWR-25-72]

This software is an implementation of the methods for initializing and training neural networks to be more efficient per parameter, described more fully below and in the related publication: In theory, depth should make a ReLU network EXPONENTIALLY more efficient by enabling it to produce an exponential number of piecewise linear sections in its output. This reasoning is largely based on the work of mathematicians that have hand-constructed networks that make good use of depth. In practice however, even very deep ReLU networks that have been randomly initialized will behave identically to their shallow counterparts - missing an entire exponential dimension of efficiency. The triangle method is a first attempt at realizing the exponential potential of deep networks. Instead of randomly setting weights, we force pairs of neurons in each layer learn to build triangles (i.e. functions from [0,1] -> [0,1] that look like triangles). This is a very efficient pattern for generating lots of linear pieces because composing two triangular functions doubles the number of pieces with each composition. The triangle method is more than just a different initialization, it is a new paradigm of training. Instead of making direct updates to the matrix weights, we do an extra step of backpropagation to collect the derivatives of the loss function with respect to the shapes of the triangles, training them to tilt left or right. This process essentially holds the networks hand throughout the loss landscape and forces it to always use depth effectively by producing triangular shapes internally. This can produce several orders of magnitude of improvement on convex one-dimensional regression problems. Much more theoretical work is needed to realize its full potential beyond this context, but the implementation in this repository will still work in arbitrary numbers of dimensions. The file Triangle_Method.py is a generalized form of the method that will build each neuron its own custom 1-d convex activation function (with exponential efficiency). Example usage on one dimensional problems can be found in Example_Usage.ipynb and an example of using this in a real neural network can be found in Example_VGG16_CIFAR10.ipynb.

Milkert, Max [National Renewable Energy Laboratory↗

Graph Metric Learning Quantifies Morphological Differences between Two Genotypes of Shoot Apical Meristem Cells in Arabidopsis

We present a method for learning “spectrally descriptive” edge weights for graphs. We generalize a previously known distance measure on graphs (Graph Diffusion Distance), thereby allowing it to be tuned to minimize an arbitrary loss function. Because all steps involved in calculating this modified GDD are differentiable, we demonstrate that it is possible for a small neural network model to learn edge weights which minimize loss. We apply this method to discriminate between graphs constructed from shoot apical meristem images of two genotypes of Arabidopsis thaliana specimens: wild-type and trm678 triple mutants with cell division phenotype. Training edge weights and kernel parameters with contrastive loss produces a learned distance metric with large margins between these graph categories. We demonstrate this by showing improved performance of a simple k-nearest-neighbors classifier on the learned distance matrix. We also demonstrate a further application of this method to biological image analysis. Once trained, we use our model to compute the distance between the biological graphs and a set of graphs output by a cell division simulator. Comparing simulated cell division graphs to biological ones allows us to identify simulation parameter regimes which characterize mutant vs. wild-type Arabidopsis cells. We find that trm678 mutant cells are characterized by increased randomness of division planes and decreased ability to avoid previous vertices between cell walls.

59 BASIC BIOLOGICAL SCIENCES↗

4DCT-based measurement of changes in pulmonary function following a course of radiation therapy

Purpose: Radiation therapy (RT) for lung cancer is commonly limited to subtherapeutic doses due to unintended toxicity to normal lung tissue. Reducing the frequency of occurrence and magnitude of normal lung function loss may benefit from treatment plans that incorporate the regional lung and radiation dose information. In this article, the authors propose a method that quantitatively measures the regional changes in lung tissue function following a course of radiation therapy by using 4DCT and image registration techniques. Methods: 4DCT data sets before and after RT from two subjects are used in this study. Nonlinear 3D image registration is applied to register an image acquired near end inspiration to an image acquired near end expiration to estimate the pulmonary function. The Jacobian of the image registration transformation, indicating local lung expansion or contraction, serves as an index of regional pulmonary function. Approximately 120 annotated vascular bifurcation points are used as landmarks to evaluate registration accuracy. The authors compare regional pulmonary function before and after RT to the planned radiation dose at different locations of the lung. Results: In all registration pairs, the average landmark distances after registration are on the order of 1 mm. The pulmonary function change as indicated by the Jacobian change ranges from -0.15 to 0.1 in the contralateral lung and -0.22 to 0.23 in the ipsilateral lung for subject A, and ranges from -0.4 to 0.39 in the contralateral lung and -0.25 to 0.5 in the ipsilateral lung for subject B. Both of the subjects show larger range of the increase in the pulmonary function in the ipsilateral lung than the contralateral lung. For lung tissue regions receiving a radiation dose larger than 24 Gy, a decrease in pulmonary function was observed. For regions receiving a radiation dose smaller than 24 Gy, either an increase or a decrease in pulmonary function was observed. The relationship between the pulmonary function change and the radiation dose varies at different locations. Conclusions: With the use of 4DCT and image registration techniques, the pulmonary function prior to and following a course of radiation therapy can be measured. In the preliminary application of this approach for two subjects, changes in pulmonary function were observed with a weak correlation between the dose and pulmonary function change. In certain sections of the lung, detected locally compromised pulmonary function may have resulted from radiation injury.

61 RADIATION PROTECTION AND DOSIMETRY↗