Energy Efficient Deep Neural Network Processing: Digital CMOS Limits and Prospects for Analog In-Memory Computing.
Abstract not provided.
Marinella, Matthew↗
SEARCH · Search NASA
Results for “COMPUTER NETWORKS”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Computable and Operationally Meaningful Multipartite Entanglement Measures
Multipartite entanglement is an essential resource for quantum communication, quantum computing, quantum sensing, and quantum networks. The utility of a quantum state |ψ$\rangle$ for these applications is often directly related to the degree or type of entanglement present in |ψ$\rangle$. Therefore, efficiently quantifying and characterizing multipartite entanglement is of paramount importance. Here, in this work, we introduce a family of multipartite entanglement measures, called concentratable entanglements. Several well-known entanglement measures are recovered as special cases of our family of measures, and hence we provide a general framework for quantifying multipartite entanglement. We prove that the entire family does not increase, on average, under local operations and classical communications. We also provide an operational meaning for these measures in terms of probabilistic concentration of entanglement into Bell pairs. Finally, we show that these quantities can be efficiently estimated on a quantum computer by implementing a parallelized SWAP test, opening up a research direction for measuring multipartite entanglement on quantum devices.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Accelerating Finite-temperature Kohn-Sham Density Functional Theory\ with Deep Neural Networks
We present a numerical modeling workflow based on machine learning (ML) which reproduces the the total energies produced by Kohn-Sham density functional theory (DFT) at finite electronic temperature to within chemical accuracy at negligible computational cost. Based on deep neural networks, our workflow yields the local density of states (LDOS) for a given atomic configuration. From the LDOS, spatially-resolved, energy-resolved, and integrated quantities can be calculated, including the DFT total free energy, which serves as the Born-Oppenheimer potential energy surface for the atoms. We demonstrate the efficacy of this approach for both solid and liquid metals and compare results between independent and unified machine-learning models for solid and liquid aluminum. Our machine-learning density functional theory framework opens up the path towards multiscale materials modeling for matter under ambient and extreme conditions at a computational scale and cost that is unattainable with current algorithms.
36 MATERIALS SCIENCE↗
In situ Parallel Training of Analog Neural Network Using Electrochemical Random-Access Memory
In-memory computing based on non-volatile resistive memory can significantly improve the energy efficiency of artificial neural networks. However, accurate in situ training has been challenging due to the nonlinear and stochastic switching of the resistive memory elements. One promising analog memory is the electrochemical random-access memory (ECRAM), also known as the redox transistor. Its low write currents and linear switching properties across hundreds of analog states enable accurate and massively parallel updates of a full crossbar array, which yield rapid and energy-efficient training. While simulations predict that ECRAM based neural networks achieve high training accuracy at significantly higher energy efficiency than digital implementations, these predictions have not been experimentally achieved. In this work, we train a 3 × 3 array of ECRAM devices that learns to discriminate several elementary logic gates (AND, OR, NAND). We record the evolution of the network’s synaptic weights during parallel in situ (on-line) training, with outer product updates. Due to linear and reproducible device switching characteristics, our crossbar simulations not only accurately simulate the epochs to convergence, but also quantitatively capture the evolution of weights in individual devices. The implementation of the first in situ parallel training together with strong agreement with simulation results provides a significant advance toward developing ECRAM into larger crossbar arrays for artificial neural network accelerators, which could enable orders of magnitude improvements in energy efficiency of deep neural networks.
97 MATHEMATICS AND COMPUTING↗
Parallel replica dynamics method for bistable stochastic reaction networks: Simulation and sensitivity analysis
Stochastic reaction networks that exhibit bistable behavior are common in systems biology, materials science, and catalysis. Sampling of stationary distributions is crucial for understanding and characterizing the long-time dynamics of bistable stochastic dynamical systems. However, simulations are often hindered by the insufficient sampling of rare transitions between the two metastable regions. In this paper, we apply the parallel replica method for a continuous time Markov chain in order to improve sampling of the stationary distribution in bistable stochastic reaction networks. The proposed method uses parallel computing to accelerate the sampling of rare transitions. Furthermore, it can be combined with the path-space information bounds for parametric sensitivity analysis. With the proposed methodology, we study three bistable biological networks: the Schlögl model, the genetic switch network, and the enzymatic futile cycle network. We demonstrate the algorithmic speedup achieved in these numerical benchmarks. More significant acceleration is expected when multi-core or graphics processing unit computer architectures and programming tools such as CUDA are employed
97 MATHEMATICS AND COMPUTING↗
Cross-Facility Orchestration of Electrochemistry Experiments and Computations
Instrument-computing ecosystems supporting automated electrochemical workflows typically require the integration of disparate instruments such as syringe pump, fraction collector, and potentiostat, all connected to an electrochemical cell. These specialized instruments with custom software and interfaces are not typically designed for network integration and remote automation. We developed a networked ecosystem of these instruments and computing platforms, which includes software to enable automated workflow orchestration from remote computers. Specifically, we developed Python wrappers of APIs and custom Pyro client-server modules to support remote operation of these instruments over the ecosystem network. Herein, we describe a specific workflow for generating and validating voltammogram (I-V) measurements of an electrolyte solution pumped into the electrochemical cell. We demonstrate the orchestration of this workflow which is composed using a Jupyter notebook and executed on a remote computer.
Al Najjar, Anees↗
Deep Neural Network High Spatiotemporal Resolution Precipitation Estimation (Deep-STEP) Using Passive Microwave and Infrared Data
Recent developments in “headline-making” deep neural networks (DNNs), specifically convolutional neural networks (CNNs), along with advancements in computational power, open great opportunities to integrate massive amounts of real-time observations to characterize spatiotemporal structures of surface precipitation. This study aims to develop a CNN algorithm, named Deep Neural Network High Spatiotemporal Resolution Precipitation Estimation (Deep-STEP), that ingests direct satellite passive microwave (PMW) brightness temperatures (Tbs) at emission and scattering frequencies combined with infrared (IR) Tbs from geostationary satellites and surface information to automatically extract geospatial features related to the precipitable clouds. These features allow the end-to-end Deep-STEP algorithm to instantaneously map surface precipitation intensities with a spatial resolution of 4 km. The main advantages of Deep-STEP, as compared to current state-of-the-art techniques, are 1) it learns and estimates complex precipitation systems directly from raw measurements in near–real time, 2) it uses the automatic spatial neighborhood feature extraction approach, and 3) it fuses coarse-resolution PMW footprints with IR images to reliably retrieve surface precipitation at a high spatial resolution. We anticipate our proposed DNN algorithm to be a starting point for more sophisticated and efficient precipitation retrieval systems in terms of accuracy, fine spatial pattern detection skills, and computational costs.
54 ENVIRONMENTAL SCIENCES↗
Complexity-calibrated benchmarks for machine learning reveal when prediction algorithms succeed and mislead
Abstract Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano’s inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least $$\sim 60\%$$ ∼ 60 % higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines—specifically, large $$\epsilon$$ ϵ -machines—are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.
97 MATHEMATICS AND COMPUTING↗
Improved pore network models to simulate single-phase flow in porous media by coupling with lattice Boltzmann method
In this paper, different pore network models to simulate single-phase flow in porous media are built and their accuracy is evaluated. In addition to the conventional pore network model (CPNM) which consists of regular pore bodies and throat bonds, three improved pore network models (IPNMs) are developed allowing to better describing the real pore and throat geometry. The first improved pore network model (IPNM1) replaces the regular throat bond with a throat bond showing the real throat cross section. The second improvement (IPNM2) uses a series of sub-throat bonds with varying cross sections to better describe the real throat geometry, which is firstly proposed in this paper. The third model (IPNM3) extracts the real pore-throat-pore geometry without simplification. The conductance of fluid flow through these more realistic throat bonds is calculated by the lattice Boltzmann method (LBM). The accuracy and computational efficiency of the different pore network models are evaluated taking the LBM simulation over the whole porous medium as reference solution. The global permeability and detailed pressure distributions in the pores for the different pore network models are validated. The results show that the accuracy of the pore network model increases from CPNM to IPNM3, but at the expense of increasing computational cost. This study suggests that IPNM3 can replace a whole-domain LBM simulation with similar accuracy but much lower computational cost. As a first-order approximation the newly proposed IPNM2 is suggested as good compromise between accuracy and computational cost.
42 ENGINEERING↗
CrossSim Inference Manual v2.0
Neural networks are largely based on matrix computations. During forward inference, the most heavily used compute kernel is the matrix-vector multiplication (MVM): $W \vec{x} $. Inference is a first frontier for the deployment of next-generation hardware for neural network applications, as it is more readily deployed in edge devices, such as mobile devices or embedded processors with size, weight, and power constraints. Inference is also easier to implement in analog systems than training, which has more stringent device requirements. The main processing kernel used during inference is the MVM.
97 MATHEMATICS AND COMPUTING↗
Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders
A common strategy for the dimensionality reduction of nonlinear partial differential equations (PDEs) relies on the use of the proper orthogonal decomposition (POD) to identify a reduced subspace and the Galerkin projection for evolving dynamics in this reduced space. However, advection-dominated PDEs are represented poorly by this methodology since the process of truncation discards important interactions between higher-order modes during time evolution. In this study, we demonstrate that encoding using convolutional autoencoders (CAEs) followed by a reduced-space time evolution by recurrent neural networks overcomes this limitation effectively. We demonstrate that a truncated system of only two latent space dimensions can reproduce a sharp advecting shock profile for the viscous Burgers equation with very low viscosities, and a six-dimensional latent space can recreate the evolution of the inviscid shallow water equations. Additionally, the proposed framework is extended to a parametric reduced-order model by directly embedding parametric information into the latent space to detect trends in system evolution. Furthermore, our results show that these advection-dominated systems are more amenable to low-dimensional encoding and time evolution by a CAE and recurrent neural network combination than the POD-Galerkin technique.
97 MATHEMATICS AND COMPUTING↗
Constrained Graph Optimization: Interdiction and Preservation Problems
The maximum flow, shortest path, and maximum matching problems are a set of basic graph problems that are critical in theoretical computer science and applications. Constrained graph optimization, a variation of these basic graph problems involving modification of the underlying graph, is equally important but sometimes significantly harder. In particular, one can explore these optimization problems with additional cost constraints. In the preservation case, the optimizer has a budget to preserve vertices or edges of a graph, preventing them from being deleted. The optimizer wants to find the best set of preserved edges/vertices in which the cost constraints are satisfied and the basic graph problems are optimized. For example, in shortest path preservation, the optimizer wants to find a set of edges/vertices within which the shortest path between two predetermined points is smallest. In interdiction problems, one deletes vertices or edges from the graph with a particular cost in order to impede the basic graph problems as much as possible (for example, delete edges/vertices to maximize the shortest path between two predetermined vertices). Applications of preservation problems include optimal road maintenance, power grid maintenance, and job scheduling, while interdiction problems are related to drug trafficking prevention, network stability assessment, and counterterrorism. Computational hardness results are presented, along with heuristic methods for approximating solutions to the matching interdiction problem. Also, efficient algorithms are presented for special cases of graphs, including on planar graphs. The graphs in many of the listed applications are planar, so these algorithms have important practical implications.
97 MATHEMATICS AND COMPUTING↗
Workshop on Incomplete Network Data Held at Sandia National Labs – Livermore
While network analysis is applied in a broad variety of scientific fields (including physics, computer science, biology, and the social sciences), how networks are constructed and the resulting bias and incompleteness have drawn more limited attention. For example, in biology, gene networks are typically developed via experiment -- many actual interactions are likely yet to be discovered. In addition to this incompleteness, the data-collection processes can introduce significant bias into the observed network datasets. For instance, if you observe part of the World Wide Web network through a classic random walk, then high degree nodes are more likely to be found than if you had selected nodes at random. Unfortunately, such incomplete and biasing data collection methods must be often used.
97 MATHEMATICS AND COMPUTING↗
Scalable multiscale modeling of platelets with 100 million particles
Here, we developed the core components of the AI-aided multiple time stepping algorithm for multiscale modeling of cell dynamics. This algorithm was implemented and analyzed on two supercomputer architectures with an application of simulating the aggregation of 250 platelets, or 102 million particles. To scale on these computers with complex memory and network architectures with GPUs, we devised a biomechanics-informed task mapping scheme to optimize load imbalance, communications, and memory utilization. Our simulations, scaling well up to 192 nodes on a Summit-like supercomputer with a peak speed of 11 petaflops, achieved a rate of 423 μs/day which is 500 times faster than the conventional algorithm using static time step and this has enabled studies of record size blood clots at record spatial–temporal resolutions. Additionally, we discovered the sensitive dependence of the scalability and execution time on the methods of decomposition, CPU–GPU coupling, and task mapping.
97 MATHEMATICS AND COMPUTING↗
Speeding up and reducing memory usage for scientific machine learning via mixed precision
Scientific machine learning (SciML) has emerged as a versatile approach to address complex computational science and engineering problems. Within this field, physics-informed neural networks (PINNs) and deep operator networks (DeepONets) stand out as the leading techniques for solving partial differential equations by incorporating both physical equations and experimental data. However, training PINNs and DeepONets require significant computational resources, including long computational times and large amounts of memory. In search of computational efficiency, training neural networks using half precision (float16) rather than the conventional single (float32) or double (float64) precision has gained substantial interest, given the inherent benefits of reduced computational time and memory consumed. However, we find that float16 cannot be applied to SciML methods, because of gradient divergence at the start of training, weight updates going to zero, and the inability to converge to a local minima. To overcome these limitations, we explore mixed precision, which is an approach that combines the float16 and float32 numerical formats to reduce memory usage and increase computational speed. Our experiments showcase that mixed precision training not only substantially decreases training times and memory demands but also maintains model accuracy. Here, we also reinforce our empirical observations with a theoretical analysis. The research has broad implications for SciML in various computational applications.
97 MATHEMATICS AND COMPUTING↗
On a theory of stability for nonlinear stochastic chemical reaction networks
We present elements of a stability theory for small, stochastic, nonlinear chemical reaction networks. Steady state probability distributions are computed with zero-information (ZI) closure, a closure algorithm that solves chemical master equations of small arbitrary nonlinear reactions. Stochastic models can be linearized around the steady state with ZI-closure, and the eigenvalues of the Jacobian matrix can be readily computed. Eigenvalues govern the relaxation of fluctuation autocorrelation functions at steady state. Autocorrelation functions reveal the time scales of phenomena underlying the dynamics of nonlinear reaction networks. In accord with the fluctuation-dissipation theorem, these functions are found to be congruent to response functions to small perturbations. Significant differences are observed in the stability of nonlinear reacting systems between deterministic and stochastic modeling formalisms.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
A Multi-scale Computational Platform to Mechanistically Assess the Effect of Genetic Variation on Drug Responses in Human Erythrocyte Metabolism
Progress in systems medicine brings promise to addressing patient heterogeneity and individualized therapies. Recently, genome-scale models of metabolism have been shown to provide insight into the mechanistic link between drug therapies and systems-level off-target effects while being expanded to explicitly include the three-dimensional structure of proteins. The integration of these molecular-level details, such as the physical, structural, and dynamical properties of proteins, notably expands the computational description of biochemical network-level properties and the possibility of understanding and predicting whole cell phenotypes. In this study, we present a multi-scale modeling framework that describes biological processes which range in scale from atomistic details to an entire metabolic network. Using this approach, we can understand how genetic variation, which impacts the structure and reactivity of a protein, influences both native and drug-induced metabolic states. As a proof-of-concept, we study three enzymes (catechol-O-methyltransferase, glucose-6-phosphate dehydrogenase, and glyceraldehyde-3-phosphate dehydrogenase) and their respective genetic variants which have clinically relevant associations. Using all-atom molecular dynamic simulations enables the sampling of long timescale conformational dynamics of the proteins (and their mutant variants) in complex with their respective native metabolites or drug molecules. We find that changes in a protein’s structure due to a mutation influences protein binding affinity to metabolites and/or drug molecules, and inflicts large-scale changes in metabolism
59 BASIC BIOLOGICAL SCIENCES↗
What do physics-informed DeepONets learn? Understanding and improving training for scientific computing applications
Physics-informed deep operator networks (DeepONets) have emerged as a promising approach toward numerically approximating the solution of partial differential equations (PDEs). In this work, we aim to develop further understanding of what is being learned by physics-informed DeepONets by assessing the universality of the extracted basis functions and demonstrating their potential toward model reduction with spectral methods. Results provide clarity about measuring the performance of a physics-informed DeepONet through the decays of singular values and expansion coefficients. In addition, we propose a transfer learning approach for improving training for physics-informed DeepONets between parameters of the same PDE as well as across different, but related, PDEs where these models struggle to train well. This approach results in significant error reduction and learned basis functions that are more effective in representing the solution of a PDE.
Deep operator networks↗