Search NASA⌕ Search

SEARCH · Search NASA

Results for “adaptive activation function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Here we propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs) for inverse problems: Determining material properties for heterogeneous systems

Here, we determine spatially varying discontinuous material properties using a domain-decomposition based physics-informed neural networks (PINNs) framework named the Adaptive Interface-PINNs or AdaI-PINNs (Roy et al., 2024). We propose the use of distinct neural networks for the field variables and material properties within each material, utilizing adaptive activation functions. While the neural networks across different materials share the same weights and biases, their activation functions are uniquely tailored using a hyperparameter that influences the slope of the activation function. The proposed framework is tested on several one-dimensional and two-dimensional benchmark examples, and its performance is compared with conventional PINNs and existing domain-decomposition PINNs frameworks, namely, the Multi-domain physics-informed neural network (M-PINN), and the eXtended physics-informed neural networks (XPINNs). The results demonstrate that the proposed approach can determine randomly distributed discontinuous material properties with an L 2 error of $\mathscr{O}$ (10 -3 ) for the material property and the root-mean-square error of $\mathscr{O}$ (10 -3 ) for the primary variable while the other approaches yield errors that are approximately two orders of magnitude larger (that is, $\mathscr{O}$ (10 -1 )). Moreover, the spatial distribution of material properties obtained using the proposed framework is in close agreement with the true distribution, whereas the other approaches fare much worse. Additionally, the proposed approach is approximately 40% faster than its competitors, indicating its potential as a robust alternative for solving inverse problems in heterogeneous materials.

36 MATERIALS SCIENCE↗
DOE OSTI2025

From PINNs to PIKANs: recent advances in physics-informed machine learning

Physics-Informed Neural Networks (PINNs) have emerged as a key tool in Scientific Machine Learning since their introduction in 2017, enabling the efficient solution of ordinary and partial differential equations using sparse measurements. Over the past few years, significant advancements have been made in the training and optimization of PINNs, covering aspects such as network architectures, adaptive refinement, domain decomposition, and the use of adaptive weights and activation functions. A notable recent development is the Physics-Informed Kolmogorov-Arnold Networks (PIKANS), which leverage a representation model originally proposed by Kolmogorov in 1957, offering a promising alternative to traditional PINNs. In this review, we provide a comprehensive overview of the latest advancements in PINNs, focusing on improvements in network design, feature expansion, optimization techniques, uncertainty quantification, and theoretical insights. We also survey key applications across a range of fields, including biomedicine, fluid and solid mechanics, geophysics, dynamical systems, heat transfer, chemical engineering, and beyond. Lastly, we review computational frameworks and software tools developed by both academia and industry to support PINN research and applications.

Kolmogorov-Arnold networks↗
DOE OSTI2024

Integrating adaptive learning with post hoc model explanation and symbolic regression to build interpretable surrogate models

Abstract We develop a materials informatics workflow to build an interpretable surrogate model for micromagnetic simulations. Our goal is to predict the energy barrier of a moving isolated skyrmion in rare-earth-free $$\hbox {Mn}_4$$ Mn 4 N. Our approach integrates adaptive learning with post hoc model explanation and symbolic regression methods. We discuss an unexplored acquisition function (information condensing active learning) within the adaptive learning loop and compare it with the known standard deviation function for efficient navigation of the search space. Model-agnostic post hoc explanation techniques then uncover trends learned by the trained model, which we then leverage to constrain the expressions used for symbolic regression. Graphical abstract

Biswas, Ankita↗
DOE OSTI2023

Extending Parsimonious Bayesian Inference

Parsimonious Bayesian inference is a theoretical framework for efficient data assimilation that seeks to balance increased consistency between predictions and training data against corresponding increases in model complexity. Within this framework, over-training is understood as optimization that encodes excessive information within model parameters while only achieving small improvements between predictions and training data. This project aims to develop practical methods of limiting excess model information during optimization. One key observation is that practical heuristics for parsimonious learning in high-dimensions must balance expressivity, i.e. the ability of the model to capture diverse predictions with only a few non-zero parameters, against discoverability, i.e. the ability to train the model with gradient-based optimization and drive parameters to low information states. As such, we developed logical activation functions that are able to adaptively approximate arbitrary truth tables that define Boolean logic operations within a probabilistic framework. These functions have demonstrated the ability to learn exclusive disjunction (XOR) and conditioned disjunction (if [condition] then [result_if_true] else [result_if_false]) within a single layer of a neural network. To efficiently exploit these activation functions to drive parsimonious learning required several other advances within the domain of variational inference. The most efficient form of complexity suppression is structured sparsification, driving most model parameters to zero while achieving the structural coherence among nonzeros needed for bandwidth reduction. Such models are not only far more efficient at suppressing information-theoretic complexity, they also reduce the other forms of complexity (computations, communication, storage, and the number of dependencies needed to evaluate predictions). Aiming to support enhanced sparsification, this project examined new approaches to high-dimensional variational inference that allow us to calibrate and control parameter uncertainty during optimization. By identifying which parameters can sustain sparsifying perturbations with little impact on prediction quality, we can develop better pruning strategies by framing them as approximate Bayesian inference. These advances also open paths to mitigate concerns with deploying advanced learning methods in resource-constrained environments, such as running models on power-limited or communication-limited devices.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Symbiotic bacteriophages exhibit multiple adaptive strategies in activated sludge flocs and contribute to floc stability

Despite the importance of phages for the dynamics and functions of microbial communities, it remains largely unexplored how symbiotic phages adapt in activated sludge systems and influence microbial aggregate stability, which is critical for clarification and reliable performance. Here, based on 12,127 phage contigs recovered from 12 activated sludge microbiomes, the symbiotic phages exhibited broad host ranges and infected dominant prokaryotes in activated sludge. Moreover, these phage communities exhibited high lysogenicity (49% to 66% lysogenic phages) and harbored diverse auxiliary metabolic genes (AMGs) that could enhance microbial aggregate stability, such as genes coding for lipopolysaccharide, peptidoglycan, and extracellular polysaccharide biosynthesis. Additionally, prokaryotic antiviral systems were widely distributed in dominant prokaryotes, particularly the Restriction-Modification (RM) and CRISPR-Cas systems, which could mitigate potential harmful phage infections. Overall, this study reveals the contemporary beneficial relationship between phages and prokaryotic hosts in activated sludge microbiomes and the potential benefits of symbiotic phages on floc stability.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-Informed Neural Networks Framework for Interface Problems

Here, we present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://doi.org/10.1016/j.cma.2024.117135), which involves domain decomposition and assignment of different predefined activation functions to the neural networks in each subdomain across a sharp interface, while keeping all other parameters of the neural networks identical. In AdaI-PINNs, the activation functions vary solely in their slopes, which are trained along with the other parameters of the neural networks. This makes the AdaI-PINNs framework fully automated without requiring preset activation functions. Comparative studies on one-dimensional, two-dimensional, and three-dimensional benchmark elliptic interface problems reveal that AdaI-PINNs outperform I-PINNs, reducing computational costs by 2-6 times while producing similar or better accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Adaptive, Active Learning, and Multifidelity Monte Carlo Methods in the MOOSE Stochastic Tools Module

MOOSE is an open-source computational platform for constructing multi-physics models and executing them in a massively parallel fashion. It has a stochastic tools module (STM) for forward/inverse uncertainty quantification (UQ) and surrogate modeling. This presentation details some recent developments to the STM with respect to the implementation of adaptive, active learning, and multifidelity Monte Carlo methods for forward UQ of computational models. Specifically, the adaptive Monte Carlo methods include Markov Chain Monte Carlo (MCMC)-driven algorithms like adaptive importance sampling and parallelized subset simulation for statistical QoI estimation, rare events analysis, and stochastic gradient-free optimization. The active learning methods include Gaussian Process (GP) surrogates and their training via Adam optimization, design of acquisition functions, and integration with samplers like Monte Carlo, adaptive importance, and parallelized subset simulation. These active learning methods are also designed to work in a batch mode, wherein, the required calls to the full computational model are executed in parallel whenever a user-specified batch size is met. The multifidelity methods in STM are broadly divided into two categories: hierarchical, where a defined hierarchy exists among the low-fidelity models, and peer, where all the low-fidelity models are treated equally. A GP surrogate is used to learn the differences between the low- and high-fidelity models in both multifidelity categories, and acquisition functions from the active learning classes are used to decide whether to rely on a low-fidelity model or call the expensive high-fidelity model. Alongside the software description and usage, applications are also presented to nuclear engineering computational models including a TRISO nuclear fuel particle, a reactor pressure vessel, and a heat-pipe microreactor.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Self-assembled reconfigurable pump architectures via magnetic colloidal swarms

Self-assembled swarms of interactive active units, which are adaptive and dynamically reconfigurable to accommodate different functionalities, represent a promising platform for the development of next-generation robotics. Here, we utilize the emergent collective behavior of active magnetic colloids confined in quasi-two-dimensional arrays of overlapping wells to demonstrate the self-organization of a colloidal swarm into a dynamic pump architecture capable of controlled transport of passive cargo particles. This dynamic architecture provides a global unidirectional looping flow pattern along the entire length of the system. We show that the flow direction of the dynamic swarm-based pump can be externally controlled by a phase shift of a driving magnetic field energizing the swarm. The experimental observations are supported by computational modeling based on phenomenological coarse-grained particle dynamics coupled to shallow-water Navier-Stokes hydrodynamics. In conclusion, our findings demonstrate how the emergent collective behavior of a swarm can be orchestrated into a desired functionality by exploiting the interplay between activity and confinement potentials.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Integrating subsystem embedding subalgebras and coupled cluster Green’s function: a theoretical foundation for quantum embedding in excitation manifold

Here, in this study, we introduce a novel approach to coupled-cluster Green's function (CCGF) embedding by seamlessly integrating conventional CCGF theory with the state-of-the-art sub-system embedding sub-algebras coupled cluster (SES-CC) formalism. This integration focuses primarily on delineating the characteristics of the sub-system and the corresponding segments of the Green's function, defined explicitly by active orbitals. Crucially, our work involves the adaptation of the SES-CC paradigm, addressing the left eigenvalue problem through a distinct form of Hamiltonian similarity transformation. This advancement not only facilitates a comprehensive representation of the interaction between the embedded sub-system and its surrounding environment but also paves the way for the quantum mechanical description of multiple embedded domains, particularly by employing the emergent quantum flow algorithms. Our theoretical underpinnings further set the stage for a generalization to multiple embedded sub-systems. This expansion holds significant promise for the exploration and application of non-equilibrium quantum systems, enhancing the understanding of system–environment interactions. In doing so, the research underscores the potential of SES-CC embedding within the realm of quantum computations and multi-scale simulations, promising a good balance between accuracy and computational efficiency.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs) for transient diffusion: Applications to forward and inverse problems in heterogeneous media

We model transient diffusion in heterogeneous materials using a novel physics-informed neural networks framework (PINNs) termed Adaptive interface physics-informed neural networks or AdaI-PINNs (Roy et al. arXiv preprint arXiv:2406.04626, 2024). AdaI-PINNs utilize different activation functions with trainable slopes tailored to each material region within the computational domain, allowing for a fully automated and adaptive PINNs approach to model interface problems with strongly and weakly discontinuous solutions. To enhance its performance in highly heterogeneous transient diffusion systems, we prescribe a suite of robust practices, including appropriate non-dimensionalization of equations, a biased sampling method, Glorot initialization, and the hard enforcement of boundary and initial conditions. Here we evaluate the efficacy of the proposed method on several benchmark forward and inverse problems. Comparative studies on one-dimensional and two-dimensional benchmark problems reveal that the modified AdaI-PINNs outperform its unmodified counterpart, achieving root-mean-square errors that are at least two orders of magnitude better in forward problems. For inverse problems, the maximum errors in the approximated diffusion coefficients by modified AdaI-PINNs are four orders of magnitude better than those of the unmodified version. Additionally, modified AdaI-PINNs demonstrate improved stability in problems with large material mismatches.

42 ENGINEERING↗
DOE OSTI2024

The crystal structure of Grindelia robusta 7,13-copalyl diphosphate synthase reveals active site features controlling catalytic specificity

Diterpenoid natural products serve critical functions in plant development and ecological adaptation and many diterpenoids have economic value as bioproducts. The family of class II diterpene synthases catalyzes the committed reactions in diterpenoid biosynthesis, converting a common geranylgeranyl diphosphate precursor into different bicyclic prenyl diphosphate scaffolds. Enzymatic rearrangement and modification of these precursors generate the diversity of bioactive diterpenoids. We report the crystal structure of Grindelia robusta 7,13-copalyl diphosphate synthase, GrTPS2, at 2.1 Å of resolution. GrTPS2 catalyzes the committed reaction in the biosynthesis of grindelic acid, which represents the signature metabolite in species of gumweed (Grindelia spp., Asteraceae). Grindelic acid has been explored as a potential source for drug leads and biofuel production. The GrTPS2 crystal structure adopts the conserved three-domain fold of class II diterpene synthases featuring a functional active site in the γβ-domain and a vestigial α-domain. Substrate docking into the active site of the GrTPS2 apo protein structure predicted catalytic amino acids. Biochemical characterization of protein variants identified residues with impact on enzyme activity and catalytic specificity. Specifically, mutagenesis of Y457 provided mechanistic insight into the position-specific deprotonation of the intermediary carbocation to form the characteristic 7,13 double bond of 7,13-copalyl diphosphate.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Mechanically activated and deactivated ion transport across nanopores with heterogeneous surface charge distributions

To mimic the intricate and adaptive functionalities of biological ion channels, electrohydrodynamic ion transport has been studied extensively, albeit mostly, across uniformly charged nanochannels. Here, we analyze the ion transport under coupled electric field and pressure across heterogeneously charged nanopores with oppositely charged sections on their lateral surface. We only consider such pores with symmetric hourglass-like and cylindrical shapes to focus on the effects of the non-uniform surface charge distribution. Finite-element simulations of a continuum model demonstrate that a pressure applied in either direction of the pore-axis equally suppresses or amplifies the ionic conductance, depending on the electric field polarity, by distorting the quasi-static distribution of ions in the pore. The resulting anomalous mechanical deactivation and activation of ionic current under opposite voltage biases exhibit the functional modularity of our setup, while their intensities are highly tunable, substantially greater than those of analogous behaviors in other nanochannels, and fundamentally correlated to ionic current rectification (ICR) in our pores. A detailed study of ICR subsequently reveals its counterintuitive non-monotonous variations, in the pores, with the magnitude of applied voltage and the pore length, that can help optimize their diode-like behavior. We further illustrate that while the hourglass-shaped nanopores yield the more efficient mechanical suppressors of ion transport, their cylindrical analogs are the superior rectifiers and mechanical amplifiers of ion conduction. Therefore, this article provides a blueprint for the strategic design of nanofluidic circuits to attain a robust, modular, and tunable control of ion transport under external electrical and mechanical stimuli.

Physics↗
DOE OSTI2024

Polymer Waste Valorization into Advanced Carbon Nanomaterials for Potential Energy and Environment Applications

The rise in universal population and accompanying demands have directed toward an exponential surge in the generation of polymeric waste. The estimate predicts that world-wide plastic production will rise to ≈590 million metric tons by 2050, whereas 5000 million more tires will be routinely abandoned by 2030. Handling this waste and its detrimental consequences on the Earth's ecosystem and human health presents a significant challenge. Converting the wastes into carbon-based functional materials viz. activated carbon, graphene, and nanotubes is considered the most scientific and adaptable method. Herein, this world provides an overview of the various sources of polymeric wastes, modes of build-up, impact on the environment, and management approaches. Update on advances and novel modifications made in methodologies for converting diverse types of polymeric wastes into carbon nanomaterials over the last 5 years are given. A remarkable focus is made to comprehend the applications of polymeric waste-derived carbon nanomaterials (PWDCNMs) in the CO 2 capture, removal of heavy metal ions, supercapacitor-based energy storage and water splitting with an emphasis on the correlation between PWDCNMs' properties and their performances. In conclusion, this review offers insights into emerging developments in the upcycling of polymeric wastes and their applications in environment and energy.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Heterologous expression of a fully active Azotobacter vinelandii nitrogenase Fe protein in Escherichia coli

ABSTRACT The functional versatility of the Fe protein, the reductase component of nitrogenase, makes it an appealing target for heterologous expression, which could facilitate future biotechnological adaptations of nitrogenase-based production of valuable chemical commodities. Yet, the heterologous synthesis of a fully active Fe protein of Azotobacter vinelandii ( Av NifH) in Escherichia coli has proven to be a challenging task. Here, we report the successful synthesis of a fully active Av NifH protein upon co-expression of this protein with Av IscS/U and Av NifM in E. coli . Our metal, activity, electron paramagnetic resonance, and X-ray absorption spectroscopy/extended X-ray absorption fine structure (EXAFS) data demonstrate that the heterologously expressed Av NifH protein has a high [Fe 4 S 4 ] cluster content and is fully functional in nitrogenase catalysis and assembly. Moreover, our phylogenetic analyses and structural predictions suggest that Av NifM could serve as a chaperone and assist the maturation of a cluster-replete Av NifH protein. Given the crucial importance of the Fe protein for the functionality of nitrogenase, this work establishes an effective framework for developing a heterologous expression system of the complete, two-component nitrogenase system; additionally, it provides a useful tool for further exploring the intricate biosynthetic mechanism of this structurally unique and functionally important metalloenzyme. IMPORTANCE The heterologous expression of a fully active Azotobacter vinelandii Fe protein (AvNifH) has never been accomplished. Given the functional importance of this protein in nitrogenase catalysis and assembly, the successful expression of AvNifH in Escherichia coli as reported herein supplies a key element for the further development of heterologous expression systems that explore the catalytic versatility of the Fe protein, either on its own or as a key component of nitrogenase, for nitrogenase-based biotechnological applications in the future. Moreover, the “clean” genetic background of the heterologous expression host allows for an unambiguous assessment of the effect of certain nif-encoded protein factors, such as AvNifM described in this work, in the maturation of AvNifH, highlighting the utility of this heterologous expression system in further advancing our understanding of the complex biosynthetic mechanism of nitrogenase.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Multiobjective Hyperparameter Optimization for Deep Learning Interatomic Potential Training Using NSGA-II

Deep neural network (DNN) potentials are an emerging tool for simulation of dynamical atomistic systems, with the promise of quantum mechanical accuracy at speedups of 10000$\times$. As with other DNN methods, hyperparameters used during training can make a substantial difference in model accuracy, and optimal settings vary with dataset. To enable rapid tuning of hyperparameters for DNN potential training, we developed a scalable multiobjective optimization evolutionary algorithm for supercomputers and tested it on the Summit system at the Oak Ridge Leadership Computing Facility (OLCF). The multiobjective approach is required due to the coupling of two learned values defining the potential: the energy and force. Using a large-scale implementation of the NSGA-II algorithm adapted for training DNN potentials, we discovered several optimal multiobjective combinations, including best choices of activation functions, learning rate scaling scheme, and pairing of the two radial cutoffs used in the three dimensional descriptor function.

Coletti, Mark↗
DOE OSTI2023

Domain Aware Deep-learning Algorithms Integrated with Scientific-computing Technologies (DADAIST)

This technical report summarized the contribution of the DADAIST project funded by the Data Model Convergence Initiative via the Laboratory Directed Research and Development (LDRD) investments at Pacific Northwest National Laboratory (PNNL). Specifically, we report the development of the NeuroMANCER (Neural Modules with Adaptive Nonlinear Constraints and Efficient Regularizations), a new open-source Scientific Machine Learning library for formulating and solving parametric constrained optimization problems, physics-informed system identification, and parametric optimal control problems. NeuroMANCER is using differentiable programming to combine modern data-driven models and optimization modeling language into a coherent algorithmic and software framework. NeuroMANCER is a Pytorch-based framework and adopts much of its philosophy focused on research and development, rapid prototyping, and streamlined deployment. Strong emphasis is given to extensibility, interoperability with the PyTorch ecosystem, and quick adaptability to custom domain problems. Neuromancer repository contains a comprehensive library of differentiable modules, including custom activation functions, matrix factorizations, deep learning architectures, neural differential equations, differential equation solvers, implicit layers such as iterative solvers, high-level API for symbolic expressions, API for modeling and control of dynamical systems, and extensive set of tutorial code examples in the form of python scripts and jupyter notebooks.

97 MATHEMATICS AND COMPUTING↗