Search NASA⌕ Search

SEARCH · Search NASA

Results for “activation function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Logical Activation Functions v.1.1

SAND2024-01501O Logical Activation Functions software is a PyTorch implementation from the paper, "Logical Activation Functions for Training Arbitrary Probabilistic Boolean Logic." The activation functions approximate logit-space marginalization of probabilistic truth tables from probabilistic interpretations of inputs. They also provide a general methodology to approximate logical relationships between abstract antecedents and consequents for machine learning architectures. They do not target any specific application or use-case. By training probabilistic truth tables, these activation functions can capture more expressive relationships in a neural network than typical elementwise activation functions. This code is only designed for a single compute node with a GPU and is limited to machine learning architectures than can fit within the memory of a single GPU. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Duersch, Jed↗
DOE OSTI2021

Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Here we propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

SineKAN: Kolmogorov-Arnold Networks using sinusoidal activation functions

Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed.

Reinhardt, Eric↗
DOE OSTI2021

Microbiota functional activity biosensors for characterizing nutrient metabolism in vivo

Methods for measuring gut microbiota biochemical activities in vivo are needed to characterize its functional states in health and disease. To illustrate one approach, an arabinan-containing polysaccharide was isolated from pea fiber, its structure defined, and forward genetic and proteomic analyses used to compare its effects, versus unfractionated pea fiber and sugar beet arabinan, on a human gut bacterial strain consortium in gnotobiotic mice. We produced ‘Microbiota Functional Activity Biosensors’ (MFABs) consisting of glycans covalently linked to the surface of fluorescent paramagnetic microscopic glass beads. Three MFABs, each containing a unique glycan/fluorophore combination, were simultaneously orally gavaged into gnotobiotic mice, recovered from their intestines, and analyzed to directly quantify bacterial metabolism of structurally distinct arabinans in different human diet contexts. Colocalizing pea-fiber arabinan and another polysaccharide (glucomannan) on the bead surface enhanced in vivo degradation of glucomannan. MFABs represent a potentially versatile platform for developing new prebiotics and more nutritious foods.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Interestingly, periodic functions are identified as a key component for most of the discovered AFs, which rarely exist in human designed AFs. Our method offers a novel approach for designing general and application-specific BNN architecture.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

The role of thermodynamic features on the functional activity of electron bifurcating enzymes

Electron bifurcation is a biological mechanism to drive a thermodynamically unfavorable redox reaction through direct coupling with an exergonic reaction. This process allows microorganisms to generate high energy reducing equivalents in order to sustain life and is often found in anaerobic metabolism, where the energy economy of the cell is poor. Recent work has revealed details of the redox energy landscapes for a variety of electron bifurcating enzymes, greatly expanding the understanding of how energy is transformed by this unique mechanism. Here we highlight the plasticity of these emerging landscapes, what is known regarding their mechanistic underpinnings, and provide a context for interpreting their biochemical activity within the physiological framework. Here, we conclude with an outlook for propelling the field toward an integrative understanding of the impact of electron bifurcation.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

In Vitro Encapsulation of Functionally Active Abiotic Photosensitizers Inside a Bacterial Microcompartment Shell

Bacterial microcompartments (BMCs) are self-assembling, selectively permeable protein shells that encapsulate enzymes to enhance catalytic efficiency of segments of metabolic pathways through means of confinement. The modular nature of BMC shells' structure and assembly enables programming of shell permeability and underscores their promise in biotechnology engineering efforts for applications in industry, medicine, and clean energy. Realizing this potential requires methods for encapsulation of abiotic molecules, which have been developed here for the first time. We report in vitro cargo loading of BMC shells with ruthenium photosensitizers (RuPS) by two approaches-one involving site-specific covalent labeling and the other driven by diffusion, requiring no specific interactions between cargo molecules and shell proteins. The highly stable shells retain encapsulated cargo over 1 week without egress and preserve RuPS photophysical activity. Finally, this study is an important foundation for further work that will converge biological BMC architecture with synthetic chemistry to facilitate biohybrid photocatalysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Systems and methods for dynamically reconfigurable artificial synapses and neurons with tunable activation functions

An ionic redox transistor comprises a solid channel, a solid reservoir layer, and a solid electrolyte layer disposed between the channel and the reservoir layer. The channel exhibits a substantially linear current-voltage relationship in a first range of voltages, and a nonlinear current-voltage relationship in a second range of voltages that is greater than the first range of voltages. One or both of the substantially linear current-voltage relationship or the nonlinear current-voltage relationship of the channel is varied by changing the concentration of ions such as oxygen vacancies in the channel. Ion or vacancy transport between the channel and the reservoir layer across the electrolyte layer occurs in response to applying a voltage between the channel and the reservoir layer. Subject to the first range of voltages, the channel can function as a synapse device. Subject to the second range of voltages, the channel can function as a neuron device.

Talin, Albert Alec↗
DOE OSTI2023

A unified and constructive framework for the universality of neural networks

Abstract One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal approximation property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive and elementary framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activation functions including most of the existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is as follows: any nAI activation function is universal in the space of continuous functions on compacta. It turns out that most of the existing activation functions are nAI, and thus universal. The framework induces several advantages over the contemporary counterparts. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is one of the first unified and constructive attempts that is valid for most of the existing activation functions. Third, it provides new proofs for most activation functions. Fourth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with a predetermined number of neurons and the values of weights/biases. Fifth, the framework allows us to abstractly present the first universal approximation with a favorable non-asymptotic rate. Sixth, our framework also provides insights into the developments, and hence providing constructive derivations, of some of the existing approaches.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs) for inverse problems: Determining material properties for heterogeneous systems

Here, we determine spatially varying discontinuous material properties using a domain-decomposition based physics-informed neural networks (PINNs) framework named the Adaptive Interface-PINNs or AdaI-PINNs (Roy et al., 2024). We propose the use of distinct neural networks for the field variables and material properties within each material, utilizing adaptive activation functions. While the neural networks across different materials share the same weights and biases, their activation functions are uniquely tailored using a hyperparameter that influences the slope of the activation function. The proposed framework is tested on several one-dimensional and two-dimensional benchmark examples, and its performance is compared with conventional PINNs and existing domain-decomposition PINNs frameworks, namely, the Multi-domain physics-informed neural network (M-PINN), and the eXtended physics-informed neural networks (XPINNs). The results demonstrate that the proposed approach can determine randomly distributed discontinuous material properties with an L 2 error of $\mathscr{O}$ (10 -3 ) for the material property and the root-mean-square error of $\mathscr{O}$ (10 -3 ) for the primary variable while the other approaches yield errors that are approximately two orders of magnitude larger (that is, $\mathscr{O}$ (10 -1 )). Moreover, the spatial distribution of material properties obtained using the proposed framework is in close agreement with the true distribution, whereas the other approaches fare much worse. Additionally, the proposed approach is approximately 40% faster than its competitors, indicating its potential as a robust alternative for solving inverse problems in heterogeneous materials.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-Informed Neural Networks Framework for Interface Problems

Here, we present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://doi.org/10.1016/j.cma.2024.117135), which involves domain decomposition and assignment of different predefined activation functions to the neural networks in each subdomain across a sharp interface, while keeping all other parameters of the neural networks identical. In AdaI-PINNs, the activation functions vary solely in their slopes, which are trained along with the other parameters of the neural networks. This makes the AdaI-PINNs framework fully automated without requiring preset activation functions. Comparative studies on one-dimensional, two-dimensional, and three-dimensional benchmark elliptic interface problems reveal that AdaI-PINNs outperform I-PINNs, reducing computational costs by 2-6 times while producing similar or better accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Synthesis and Exploratory Catalysis of 3d Metals: Atom and Group-Transfer Reactions and the Activation and Functionalization of Small Molecules Including Greenhouse Gases

Determining ways to convert natural gas, with zero emissions, and to more value-added materials such as olefins is one of the main goals in my research group. Over this funding period, we explored the chemistry of early-transition metals with metal-nitrogen multiple bonds, specifically titanium and zirconium nitrides, and explored their redox properties, basicity and reactivity with small molecules including greenhouses gases such as carbon dioxide. Some of these work serve as inspiration for the chemistry of important materials such as uranium nitride along with its unprecedented basicity and ability to activate C-H bonds. Using robust chelating templates we also examined rare examples of trivalent group 4 transition metal ions, and studied these spectroscopically. Using early transition metal ions we also explored synthetic routes to new phosphide based products using the phosphaethynolate salt. In addition, we also explored the chemistry of ferrous systems, and redox active ligand that can allow us to reversibly break and make C-S as well as N-N bonds, activate dinitrogen forming unusual MNNM topologies. Our final component describes how we activate methane and dehydrocouple if with a carbene source to form ethylene. Using this information, we also discovered a simple to make iridium catalyst that activate and functionalize methane with 9:1 selectivity for mono-functionalization. We have established a pathway that leads to poisoning of the catalyst and have found optimal conditions for higher selectivity and with over 170 Turnovers.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Integration of Ag-CBRAM crossbars and Mott ReLU neurons for efficient implementation of deep neural networks in hardware

In-memory computing with emerging non-volatile memory devices (eNVMs) has shown promising results in accelerating matrix-vector multiplications. However, activation function calculations are still being implemented with general processors or large and complex neuron peripheral circuits. Here, we present the integration of Ag-based conductive bridge random access memory (Ag-CBRAM) crossbar arrays with Mott rectified linear unit (ReLU) activation neurons for scalable, energy and area-efficient hardware (HW) implementation of deep neural networks. We develop Ag-CBRAM devices that can achieve a high ON/OFF ratio and multi-level programmability. Compact and energy-efficient Mott ReLU neuron devices implementing ReLU activation function are directly connected to the columns of Ag-CBRAM crossbars to compute the output from the weighted sum current. We implement convolution filters and activations for VGG-16 using our integrated HW and demonstrate the successful generation of feature maps for CIFAR-10 images in HW. Our approach paves a new way toward building a highly compact and energy-efficient eNVMs-based in-memory computing system.

Mott insulators↗
DOE OSTI2021

Whetstone v.0.9.2

Whetstone is a deep learning software library for spiking/binary threshold neuromorphic hardware. Built to integrate with Keras models, Whetstone provides a collection of layers, callbacks, and utility functions designed to allow the training of integrate-and-fire neurons rather than traditional analog activation functions (e.g. rectified linear units). Generally, training a network for neuromorphic hardware is challenging due to the discontinuity of the activation function. Whetstone overcomes this challenge by using ever-closer continuous approximations of a threshold activation, modified during training time. This allows rapid and consistent training for ultra-low powered neuromorphic hardware.

Severa, William↗