Search NASA⌕ Search

SEARCH · Search NASA

Results for “Loss function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics

Traditional data-driven deep learning models often struggle with high training costs, error accumulation, and poor generalizability in complex physical processes. Physics-informed deep learning (PiDL) addresses these challenges by incorporating physical principles into the model. Most PiDL approaches regularize training by embedding governing equations into the loss function, yet this depends heavily on extensive hyperparameter tuning to weigh each loss term. To this end, we propose to leverage physics prior knowledge by “baking” the discretized governing equations into the neural network architecture via the connection between the partial differential equations (PDE) operators and network structures, resulting in a PDE-preserved neural network (PPNN). This method, embedding discretized PDEs through convolutional residual networks in a multi-resolution setting, largely improves the generalizability and long-term prediction accuracy, outperforming conventional black-box models. The effectiveness and merit of the proposed methods have been demonstrated across various spatiotemporal dynamical systems governed by spatiotemporal PDEs, including reaction-diffusion, Burgers’, and Navier-Stokes equations.

97 MATHEMATICS AND COMPUTING↗

Prediction of laser beam spatial profiles in a high-energy laser facility by use of deep learning

We adapt the significant advances achieved recently in the field of generative artificial intelligence/machine-learning to laser performance modeling in multipass, high-energy laser systems with application to high-shot-rate facilities relevant to inertial fusion energy. Advantages of neural-network architectures include rapid prediction capability, data-driven processing, and the possibility to implement such architectures within future low-latency, low-power consumption photonic networks. Four models were investigated that differed in their generator loss functions and utilized the U-Net encoder/decoder architecture with either a reconstruction loss alone or combined with an adversarial network loss. We achieved inference times of 1.3 ms for a 256 × 256 pixel near-field beam with errors in predicted energy of the order of 1% over most of the energy range. It is shown that prediction errors are significantly reduced by ensemble averaging the models with different weight initializations. These results suggest that including the temporal dimension in such models may provide accurate, real-time spatiotemporal predictions of laser performance in high-shot-rate laser systems.

47 OTHER INSTRUMENTATION↗

Learning nonlinear level sets for dimensionality reduction in function approximation

We developed a Nonlinear Level-set Learning (NLL) method for dimensionality reduction in high-dimensional function approximation with small data. This work is motivated by a variety of design tasks in real-world engineering applications, where practitioners would replace their computationally intensive physical models (e.g., high-resolution fluid simulators) with fast-to-evaluate predictive machine learning models, so as to accelerate the engineering design processes. There are two major challenges in constructing such predictive models: (a) high-dimensional inputs (e.g., many independent design parameters) and (b) small training data, generated by running extremely time-consuming simulations. Thus, reducing the input dimension is critical to alleviate the over-fitting issue caused by data insufficiency. Existing methods, including sliced inverse regression and active subspace approaches, reduce the input dimension by learning a linear coordinate transformation; our main contribution is to extend the transformation approach to a nonlinear regime. Specifically, we exploit reversible networks (RevNets) to learn nonlinear level sets of a high-dimensional function and parameterize its level sets in low-dimensional spaces. A new loss function was designed to utilize samples of the target functions' gradient to encourage the transformed function to be sensitive to only a few transformed coordinates. The NLL approach is demonstrated by applying it to three 2D functions and two 20D functions for showing the improved approximation accuracy with the use of nonlinear transformation, as well as to an 8D composite material design problem for optimizing the buckling-resistance performance of composite shells of rocket inter-stages.

Zhang, Guannan↗

Full potential study of the elastic, electronic, and optical properties of spinels MgIn{sub 2}S{sub 4} and CdIn{sub 2}S{sub 4} under pressure effect

The structural, elastic, electronic, and optical properties of cubic spinel MgIn{sub 2}S{sub 4} and CdIn{sub 2}S{sub 4} compounds have been calculated using a full relativistic version of the full-potential linearized-augmented plane wave with the mixed basis FP/APW+lo method. The exchange and correlation potential is treated by the generalized-gradient approximation (GGA). Moreover, the Engel-Vosko GGA formalism is also applied to optimize the corresponding potential for band structure calculations. The ground state properties, including the lattice constants, the internal parameter, the bulk modulus, and the pressure derivative of the bulk modulus are in reasonable agreement with the available data. Using the total energy-strain technique, we have determined the full set of first-order elastic constants C{sub ij} and their pressure dependence, which have not been calculated or measured yet. The shear modulus, Young's modulus, and Poisson's ratio are calculated for polycrystalline XIn{sub 2}S{sub 4} aggregates. The Debye temperature is estimated from the average sound velocity. Electronic band structures show a direct band gap ({Gamma}-{Gamma}) for MgIn{sub 2}S{sub 4} and an indirect band gap (K-{Gamma}) for CdIn{sub 2}S{sub 4}. The calculated band gaps with EVGGA show a significant improvement over the GGA. The optical constants, including the dielectric function {epsilon}({omega}), the refractive index n({omega}), the reflectivity R({omega}), and the energy loss function L({omega}) were calculated for radiation up to 30 eV. -- Graphical abstract: Calculated total and partial densities of states for MgIn{sub 2}S{sub 4} and CdIn{sub 2}S{sub 4}

36 MATERIALS SCIENCE↗

Data-driven Vulnerability Analysis of Networked Pipeline System

This paper introduces an attack generation framework for evaluating the vulnerability of nonlinear networked pipeline systems. The vulnerability analysis is formulated as determining the presence of feasible attack sets, defined by boundary functions representing the effectiveness and stealthiness of attack signals with respect to the objective and attack detection module. The framework utilizes three data-driven models, including two discriminative models that learn the boundary functions and a generative model that produces elements of the feasible attack set. A new loss function ensures successful attack generation with high probability.

03 NATURAL GAS↗

Learning effective stochastic differential equations from microscopic simulations: Linking stochastic numerics to deep learning

We identify effective stochastic differential equations (SDEs) for coarse observables of fine-grained particle- or agent-based simulations; these SDEs then provide useful coarse surrogate models of the fine scale dynamics. We approximate the drift and diffusivity functions in these effective SDEs through neural networks, which can be thought of as effective stochastic ResNets. The loss function is inspired by, and embodies, the structure of established stochastic numerical integrators (here, Euler–Maruyama and Milstein); our approximations can thus benefit from backward error analysis of these underlying numerical schemes. They also lend themselves naturally to “physics-informed” gray-box identification when approximate coarse models, such as mean field equations, are available. Existing numerical integration schemes for Langevin-type equations and for stochastic partial differential equations can also be used for training; we demonstrate this on a stochastically forced oscillator and the stochastic wave equation. Our approach does not require long trajectories, works on scattered snapshot data, and is designed to naturally handle different time steps per snapshot. We consider both the case where the coarse collective observables are known in advance, as well as the case where they must be found in a data-driven manner.

97 MATHEMATICS AND COMPUTING↗

Imaging nanoscale carrier, thermal, and structural dynamics with time-resolved and ultrafast electron energy-loss spectroscopy

Time-resolved and ultrafast electron energy-loss spectroscopy (EELS) is an emerging technique for measuring photoexcited carriers, lattice dynamics, and near-fields across femtosecond to microsecond timescales. When performed in either a specialized scanning transmission electron microscope or ultrafast electron microscope (UEM), time-resolved and ultrafast EELS can directly image charge carriers, lattice vibrations, and heat dissipation following photoexcitation or applied bias. Yet, recent advances in theoretical calculations and electron optics are often required to realize the full potential of ultrafast EEL spectrum imaging. Here, in this review, we present a comprehensive overview of the recent progress in the theory and instrumentation of time-resolved and ultrafast EELS. We begin with an introduction to the technique, followed by a physical description of the loss function. We outline approaches for calculating and interpreting ground-state and transient EEL spectra spanning low-loss plasmons to core-level excitations analogous to x-ray absorption. We then survey the current state of time-resolved and ultrafast EELS techniques beyond photon-induced near-field electron microscopy, highlighting abilities to image carrier and thermal dynamics. Finally, we examine future directions enabled by emerging technologies, including electron beam monochromation, in situ and operando cells, laser-free UEM, and high-speed direct electron detectors. These advances position time-resolved and ultrafast EELS as a critical tool for uncovering nanoscale dynamic processes in quantum materials and solar energy conversion devices.

Computational methods↗

Deep-learning methods for contrast enhancement and artifact reduction in cryo-electron tomography: a systematic analysis of the state of the art and proposed improvements

Cryo-electron tomography (cryo-ET) has emerged as the preferred technique for visualizing the organization of macromolecular complexes in situ and resolving their structures at subnanometre resolution [Tegunov et al. (2021)View full citation, Nat. Methods, 18, 186–193]. Despite improvements in data quality as a result of advances in detector technology, microscope stability and stage precision, the analysis and interpretation of tomograms remains challenging due to a low signal-to-noise ratio and reconstruction artifacts stemming from experimental constraints in specimen tilt during data collection resulting in a missing wedge in the Fourier space. Recently, self-supervised deep-learning methods have been proposed for contrast enhancement and reduction of resolution anisotropy in reconstructed tomograms. Here, we evaluate several state-of-the-art deep-learning methods which aim to improve the interpretability of cryo-ET reconstructions, with a focus on their performance on downstream tasks of template matching, sub­tomogram averaging and segmentation. We propose new training architectures and a loss function based on Fourier shell correlation that show improved performance over the standard U-Net with L1/L2 losses. We demonstrate our analysis on four diverse experimental datasets: purified 80S ribosomes, in situ Chlamydomonas reinhardtii, immature HIV-1 virus-like particles and INS-1E cells.

contrast enhancement↗

Generative Vulnerability Assessment for Cyber-Physical Systems

Cyber-physical systems (CPS) are highly susceptible to malicious attacks due to their complex dynamics and interconnectivity. A comprehensive understanding of their vulnerabilities is essential for designing effective resilience measures. This paper presents a data-driven attack generative system for evaluating the vulnerability of CPS. The proposed approach formulates the vulnerability assessment problem as determining the feasibility of a specific attack set based on two boundary functions that represent the effectiveness and stealthiness of attacks. The attack generative model is trained using a custom loss function, with two universal approximators designed to learn the effectiveness and stealthiness functions simultaneously. Theoretical results for successful generation and asymptotic convergence of the resulting training algorithm are given. As a result, the proposed approach is evaluated via numerical simulation of an IEEE 14-bus system and gas pipeline systems, demonstrating its viability in learning how to attack nonlinear CPS and identify potential vulnerabilities.

Computer systems organization↗

Analysis of the structural, electronic and optic properties of Ni doped MgSiP{sub 2} semiconductor chalcopyrite compound

The structural, electronic band structure and optic properties of the Ni doped MgSiP{sub 2} chalcopyrite compound have been performed by using first-principles method in the density functional theory (DFT) as implemented in Vienna Ab-initio Simulation Package (VASP). The generalized gradient approximation (GGA) in the scheme of Perdew, Burke and Ernzerhof (PBE) is used for the exchange and correlation functional. The present lattice constant (a) follows generally the Vegard’s law. The electronic band structure, total and partial density of states (DOS and PDOS) are calculated. We present data for the frequency dependence of imaginary and real parts of dielectric functions of Ni doped MgSiP{sub 2}. For further investigation of the optical properties the reflectivity, refractive index, extinction coefficient and electron energy loss function are also predicted. Our obtained results indicate that the lattice constants, electronic band structure and optical properties of this compound are dependent on the substitution concentration of Ni.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Collective excitations and low-energy ionization signatures of relativistic particles in silicon detectors

Abstract Solid-state detectors with a low energy threshold have several applications, including searches of non-relativistic halo dark-matter particles with sub-GeV masses. When searching for relativistic, beyond-the-Standard-Model particles with enhanced cross sections for small energy transfers, a small detector with a low energy threshold may have better sensitivity than a larger detector with a higher energy threshold. In this paper, we calculate the low-energy ionization spectrum from high-velocity particles scattering in a dielectric material. We consider the full material response including the excitation of bulk plasmons. We generalize the energy-loss function to relativistic kinematics, and benchmark existing tools used for halo dark-matter scattering against electron energy-loss spectroscopy data. Compared to calculations commonly used in the literature, such as the Photo-Absorption-Ionization model or the free-electron model, including collective effects shifts the recoil ionization spectrum towards higher energies, typically peaking around 4–6 electron-hole pairs. We apply our results to the three benchmark examples: millicharged particles produced in a beam, neutrinos with a magnetic dipole moment produced in a reactor, and upscattered dark-matter particles. Our results show that the proper inclusion of collective effects typically enhances a detector’s sensitivity to these particles, since detector backgrounds, such as dark counts, peak at lower energies.

Physics↗

Reducing the Parameter Dependency of Phase-Picking Neural Networks with Dice Loss

Training a neural network for picking seismic phase arrivals has been commonly posed as a segmentation problem. It is a highly imbalanced segmentation problem in the sense that the background vastly dominates the foreground because we are trying to pick the optimal single sample point that represents the arrival of a seismic phase in a many seconds long time window. Here, we test the Dice loss, which is a preferred loss function for highly imbalanced image segmentation problems. We show that phase-picking neural networks trained on the Dice loss behave in a binary fashion for which the prediction output is almost always either nearly 1 or nearly 0. This feature removes the strong dependence of data processing workflows on the prediction score threshold, which is an otherwise critical parameter to determine when using neural networks trained on the cross-entropy loss. When strategically used, models trained on the Dice loss can reduce the parameter dependency of machine learning-based seismic monitoring.

58 GEOSCIENCES↗

Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima.

machine leaning, Bad minima, Inverse problems, Phy↗

Efficient Training of Deep Neural Operator Networks via Randomized Sampling

Neural operators (NOs) employ deep neural networks to learn the mappings between infinitedimensional function spaces. Deep operator network (DeepONet), a popular NO architecture, has demonstrated success in the real-time prediction of complex dynamics across various scientific and engineering applications. In this work, we introduce a random sampling technique to be adopted during the training of DeepONet, aimed at improving the generalization ability of the model, while significantly reducing the computational time. The proposed approach targets the trunk network of the DeepONet model that outputs the basis functions corresponding to the spatiotemporal locations of the bounded domain on which the physical system is defined. While constructing the loss function, DeepONet training traditionally considers a uniform grid of spatiotemporal points at which all the output functions are evaluated for each iteration. This approach leads to a larger batch size, resulting in poor generalization and increased memory demands, due to the limitations of the stochastic gradient descent (SGD) optimizer. The proposed random sampling over the inputs of the trunk net mitigates these challenges, improving generalization and reducing the memory requirements during training, resulting in significant computational gains. We validate our hypothesis through three benchmark examples, demonstrating substantial reductions in training time while achieving comparable or lower overall test errors relative to the traditional training approach. Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet, offering a promising avenue for improving the framework’s performance in modeling complex physical systems.

Karumuri, Sharmila [Department of Civil & Systems ↗

Bayesian Model Calibration for Extrapolative Prediction via Gibbs Posteriors

The current standard Bayesian approach to model calibration, which assigns a Gaussian process prior to the discrepancy term, often suffers from issues of unidentifiability and computational complexity and instability. When the goal is to quantify uncertainty in physical parameters for extrapolative prediction, then there is no need to perform inference on the discrepancy term. With this in mind, we introduce Gibbs posteriors as an alternative Bayesian method for model calibration, which updates the prior with a loss function connecting the data to the parameter. The target of inference is the physical parameter value which minimizes the expected loss. We propose to tune the loss scale of the Gibbs posterior to maintain nominal frequentist coverage under assumptions of the form of model discrepancy, and present a bootstrap implementation for approximating coverage rates. Our approach is highly modular, allowing an analyst to easily encode a wide variety of such assumptions. Furthermore, we provide a principled method of combining posteriors calculated from data subsets. We apply our methods to data from an experiment measuring the material properties of tantalum.

36 MATERIALS SCIENCE↗

Loss of Drp1 function alters OPA1 processing and changes mitochondrial membrane organization

RNAi mediated loss of Drp1 function changes mitochondrial morphology in cultured HeLa and HUVEC cells by shifting the balance of mitochondrial fission and fusion towards unopposed fusion. Over time, inhibition of Drp1 expression results in the formation of a highly branched mitochondrial network along with 'bulge'-like structures. These changes in mitochondrial morphology are accompanied by a reduction in levels of Mitofusin 1 (Mfn1) and 2 (Mfn2) and a modified proteolytic processing of OPA1 isoforms, resulting in the inhibition of cell proliferation. In addition, our data imply that bulge formation is driven by Mfn1 action along with particular proteolytic short-OPA1 (s-OPA1) variants: Loss of Mfn2 in the absence of Drp1 results in an increase of Mfn1 levels along with processed s-OPA1-isoforms, thereby enhancing continuous 'fusion' and bulge formation. Moreover, bulge formation might reflect s-OPA1 mitochondrial membrane remodeling activity, resulting in the compartmentalization of cytochrome c deposits. The proteins Yme1L and PHB2 appeared not associated with the observed enhanced OPA1 proteolysis upon RNAi of Drp1, suggesting the existence of other OPA1 processing controlling proteins. Taken together, Drp1 appears to affect the activity of the mitochondrial fusion machinery by unbalancing the protein levels of mitofusins and OPA1.

60 APPLIED LIFE SCIENCES↗

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. Here, we demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Taken together, our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

97 MATHEMATICS AND COMPUTING↗

Plateau Phenomenon in Gradient Descent Training of RELU Networks: Explanation, Quantification, and Avoidance

The ability of neural networks to provide ‘best in class’ approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods,a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon.analysis is carried out in the setting of univariate ReLU networks. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. Here, the main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.

97 MATHEMATICS AND COMPUTING↗