Search NASA⌕ Search

SEARCH · Search NASA

Results for “compact neural networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Modeling the AC Power Flow Equations with Optimally Compact Neural Networks: Application to Unit Commitment

Nonlinear power flow constraints render a variety of power system optimization problems computationally intractable. Emerging research shows, however, that the nonlinear AC power flow equations can be successfully modeled using neural networks. These neural networks can be exactly transformed into mixed integer linear programs and embedded inside challenging optimization problems, thus replacing nonlinearities that are intractable for many applications with tractable piecewise linear approximations. Such approaches, though, suffer from an explosion of the number of binary variables needed to represent the neural network. Accordingly, this paper develops a technique for training an "optimally compact'' neural network, i.e., one that can represent the power flow equations with a sufficiently high degree of accuracy while still maintaining a tractable number of binary variables. We demonstrate the use of this neural network as an approximator of the nonlinear power flow equations by embedding it in the AC unit commitment problem, transforming the problem from a mixed integer nonlinear program into a more manageable mixed integer linear program. We use the 14-, 57-, and 89-bus networks as test cases and compare the AC-feasibility of commitment decisions resulting from the neural network, DC, and linearized power flow approximations. Our results show that the neural network model outperforms both the DC and linearized power flow approximations when embedded in the unit commitment problem. The neural network formulation most often selects a feasible unit commitment schedule, and furthermore, it only s

AC power flow↗
DOE OSTI2022

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Post-training model compression can reduce the inference costs of deep neural networks, but uncompressed training still consumes enormous hardware resources and energy. To enable low-energy training on edge devices, it is highly desirable to directly train a compact neural network from scratch with a low memory cost. Low-rank tensor decomposition is an effective approach to reduce the memory and computing costs of large neural networks. However, directly training low-rank tensorized neural networks is a very challenging task because it is hard to determine a proper tensor rank a priori, and the tensor rank controls both model complexity and accuracy. Here, this paper presents a novel end-to-end framework for low-rank tensorized training. We first develop a Bayesian model that supports various low-rank tensor formats (e.g., CANDECOMP/PARAFAC, Tucker, tensor-train, and tensor-train matrix) and reduces neural network parameters with automatic rank determination during training. Then we develop a customized Bayesian solver to train large-scale tensorized neural networks. Our training methods shows orders-of-magnitude parameter reduction and little accuracy loss (or even better accuracy) in the experiments. On a very large deep learning recommendation system with over 4.2 ×10 9 model parameters, our method can reduce the parameter number to 1.6 ×10 5 automatically in the training process (i.e., by 2.6 ×10 4 times) while achieving almost the same accuracy. Code is available at https://github.com/colehawkins/bayesian-tensor-rank-determination.

compact neural networks↗
DOE OSTI2022

Data reduction through optimized scalar quantization for more compact neural networks

Raw data generation for several existing and planned large physics experiments now exceeds TB/s rates, generating untenable data sets in very little time. Those data often demonstrate high dimensionality while containing limited information. Meanwhile, Machine Learning algorithms are now becoming an essential part of data processing and data analysis. Those algorithms can be used offline for post processing and post data analysis, or they can be used online for real time processing providing ultra low latency experiment monitoring. Both use cases would benefit from data throughput reduction while preserving relevant information: one by reducing the offline storage requirements by several orders of magnitude and the other by allowing ultra fast online inferencing with low complexity Machine Learning models. Moreover, reducing the data source throughput also reduces material cost, power and data management requirements. In this work we demonstrate optimized nonuniform scalar quantization for data source reduction. This data reduction allows lower dimensional representations while preserving the relevant information of the data, thus enabling high accuracy Tiny Machine Learning classifier models for online fast inferences. We demonstrate this approach with an initial proof of concept targeting the CookieBox, an array of electron spectrometers used for angular streaking, that was developed for LCLS-II as an online beam diagnostic tool. We used the Lloyd-Max algorithm with the CookieBox dataset to design an optimized nonuniform scalar quantizer. Optimized quantization lets us reduce input data volume by 69% with no significant impact on inference accuracy. When we tolerate a 2% loss on inference accuracy, we achieved 81% of input data reduction. Finally, the change from a 7-bit to a 3-bit input data quantization reduces our neural network size by 38%.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Bayesian tensorized neural networks with automatic rank selection

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to address this challenge, this paper proposes a low-rank Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. Here, we provide experimental validation on a two-layer fully connected neural network, a 6-layer CNN and a 110-layer residual neural network where our work produces 7.4x to 137x more compact neural networks directly from the training while achieving high prediction accuracy.

Low-rank tensor↗
DOE OSTI2022

Photometric identification of compact galaxies, stars, and quasars using multiple neural networks

We present MargNet, a deep learning-based classifier for identifying stars, quasars, and compact galaxies using photometric parameters and images from the Sloan Digital Sky Survey Data Release 16 catalogue. MargNet consists of a combination of convolutional neural network and artificial neural network architectures. Using a carefully curated data set consisting of 240 000 compact objects and an additional 150 000 faint objects, the machine learns classification directly from the data, minimizing the need for human intervention. MargNet is the first classifier focusing exclusively on compact galaxies and performs better than other methods to classify compact galaxies from stars and quasars, even at fainter magnitudes. This model and feature engineering in such deep learning architectures will provide greater success in identifying objects in the ongoing and upcoming surveys, such as Dark Energy Survey and images from the Vera C. Rubin Observatory.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Modeling Multi-View Impedance-Based Cross-Geometry SOH Estimator for Li-ion Batteries

Abstract: Accurately estimating battery’s State of Health (SOH) remains challenging when models must generalize across cell designs and operating conditions. Most Electrochemical Impedance Spectroscopy (EIS)-based approaches either (i) hand-engineer a few Nyquist-plot features for shallow models—fast but does not generalize across geometries—or (ii) learn directly from Nyquist plots with deep networks, which removes manual feature extraction, yet still limited to a single plot type. As a result, cross-geometry robustness and deployability on constrained Internet of Things (IoT) devices remain open problems. We propose a compact Convolutional Neural Network (CNN) (∼ 10k parameters) that takes multi-representation EIS inputs—Nyquist (real/imaginary) and phase–magnitude (|Z|/ϕ) stacked as four channels, so the model can learn complementary degradation signatures while remaining small enough for fast inference. We build a dataset from cyclic aging of two geometries (LG INR18650MJ1 cylindrical cells and LIR2032 coin cells), acquire EIS every ten cycles from 10 kHz to 10 mHz (10 points/decade), and evaluate with leave-one-cell-out testing strategy. We further study fusion vs. single-representation inputs and assess feasibility for on-device deployment (e.g., NVIDIA Jetson device). The results show that training on multiple EIS representations improves SOH estimation accuracy and cross-geometry generalization compared to single-representation models, which uses only Nyquist or phase–magnitude plots. This design targets accurate, generalizable SOH prediction without manual feature engineering while enabling practical real-time use.

Bakr, Ahmed [The University of Alabama (UA)]↗
DOE OSTI2021

Deep Learning with Quantized Neural Networks for Gravitational-wave Forecasting of Eccentric Compact Binary Coalescence

We present the first application of deep learning forecasting for binary neutron stars, neutron star–black hole systems, and binary black hole mergers that span an eccentricity range e ≤ 0.9. In this work, we train neural networks that describe these astrophysical populations, and then test their performance by injecting simulated eccentric signals in advanced Laser Interferometer Gravitational-Wave Observatory (LIGO) noise available at the Gravitational Wave Open Science Center to (1) quantify how fast neural networks identify these signals before the binary components merge; (2) quantify how accurately neural networks estimate the time to merger once gravitational waves are identified; and (3) estimate the time-dependent sky localization of these events from early detection to merger. Our findings show that deep learning can identify eccentric signals from a few seconds (for binary black holes) up to tens of seconds (for binary neutron stars) prior to merger. A quantized version of our neural networks achieves 4× reduction in model size, and up to 2.5× inference speedup. These novel algorithms may be used to facilitate time-sensitive multimessenger astrophysics observations of compact binaries in dense stellar environments.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Learning high-dimensional parametric maps via reduced basis adaptive residual networks

We propose a scalable framework for the learning of high-dimensional parametric maps via adaptively constructed residual network (ResNet) maps between reduced bases of the inputs and outputs. When just few training data are available, it is beneficial to have a compact parametrization in order to ameliorate the ill-posedness of the neural network training problem. By linearly restricting high-dimensional maps to informed reduced bases of the inputs, one can compress high-dimensional maps in a constructive way that can be used to detect appropriate basis ranks, equipped with rigorous error estimates. A scalable neural network learning framework is thus to learn the nonlinear compressed reduced basis mapping. Unlike the reduced basis construction, however, neural network constructions are not guaranteed to reduce errors by adding representation power, making it difficult to achieve good practical performance. Inspired by recent approximation theory that connects ResNets to sequential minimizing flows, we present an adaptive ResNet construction algorithm. This algorithm allows for depth-wise enrichment of the neural network approximation, in a manner that can achieve good practical performance by first training a shallow network and then adapting. We prove universal approximation of the associated neural network class for $L^2_v$ functions on compact sets. Our overall framework allows for constructive means to detect appropriate breadth and depth, and related compact parametrizations of neural networks, significantly reducing the need for architectural hyperparameter tuning. Numerical experiments for parametric PDE problems and a 3D CFD wing design optimization parametric map demonstrate that the proposed methodology can achieve remarkably high accuracy for limited training data, and outperformed other neural network strategies we compared against.

42 ENGINEERING↗
DOE OSTI2023

Experimental Observations of the Topology of Convolutional Neural Network Activations

Topological data analysis (TDA) is a branch of computational mathematics, bridging algebraic topology and data science, that provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture resulting in high-dimensional, difficult to interpret internal representations of input data. As DNNs become more ubiquitous across multiple sectors of our society, there is increasing recognition that mathematical methods are needed to aid analysts, researchers, and practitioners in understanding and interpreting how these models' internal representations relate to the final classification. In this paper we apply cutting edge techniques from TDA with the goal of gaining insight towards interpretability of convolutional neural networks used for image classification. We use two common TDA approaches to explore several methods for modeling hidden layer activations as high-dimensional point clouds, and provide experimental evidence that these point clouds capture valuable structural information about the model's process. First, we demonstrate that a distance metric based on persistent homology can be used to quantify meaningful differences between layers and discuss these distances in the broader context of existing representational similarity metrics for neural network interpretability. Second, we show that a mapper graph can provide semantic insight as to how these models organize hierarchical class knowledge at each layer. These observations demonstrate that TDA is a useful tool to help deep learning practitioners unlock the hidden structures of their models.

topological data analysis, deep learning↗
DOE OSTI2023

Error Estimates of Residual Minimization Using Neural Networks for Linear PDES

We propose an abstract framework for analyzing the convergence of least-squares methods based on residual minimization when feasible solutions are neural networks. With the norm relations and compactness arguments, we derive error estimates for both continuous and discrete formulations of residual minimization in strong and weak forms. The formulations cover recently developed physicsinformed neural networks based on strong and variational formulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Autoregressive Neural Network for Simulating Open Quantum Systems via a Probabilistic Formulation

The theory of open quantum systems lays the foundation for a substantial part of modern research in quantum science and engineering. Rooted in the dimensionality of their extended Hilbert spaces, the high computational complexity of simulating open quantum systems calls for the development of strategies to approximate their dynamics. In this Letter, we present an approach for tackling open quantum system dynamics. Using an exact probabilistic formulation of quantum physics based on positive operator-valued measure, we compactly represent quantum states with autoregressive neural networks; such networks bring significant algorithmic flexibility due to efficient exact sampling and tractable density. We further introduce the concept of string states to partially restore the symmetry of the autoregressive neural network and improve the description of local correlations. Efficient algorithms have been developed to simulate the dynamics of the Liouvillian superoperator using a forward-backward trapezoid method and find the steady state via a variational formulation. Our approach is benchmarked on prototypical one-dimensional and two-dimensional systems, finding results which closely track the exact solution and achieve higher accuracy than alternative approaches based on using Markov chain Monte Carlo method to sample restricted Boltzmann machines. Our Letter provides general methods for understanding quantum dynamics in various contexts, as well as techniques for solving high-dimensional probabilistic differential equations in classical setups.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Graph-based Compact Modeling (GCM) of CMOS transistors for efficient parameter extraction: A machine learning approach

Parameter extraction of compact transistor models is an expensive process, heavily relying on engineering knowledge and experience. To automate such a process, we propose a novel approach, Graph-based Compact Model (GCM), that integrates physical modeling and data-driven learning. GCM utilizes Graph Neural Networks (GNNs) to establish the model structure, while retaining the physicality in compact models. Here, we implement our GCM in Verilog-A to support circuit simulations. As demonstrated with an academic 7 nm FinFET PDK, the new approach automatically generates a GCM model within a minute, and achieves excellent accuracy and efficiency in SPICE.

42 ENGINEERING↗
DOE OSTI2021

Energy-efficient Mott activation neuron for full-hardware implementation of neural networks

To circumvent the von Neumann bottleneck, substantial progress has been made towards in-memory computing with synaptic devices. However, compact nanodevices implementing non-linear activation functions are required for efficient full-hardware implementation of deep neural networks. Here, in this work, we present an energy-efficient and compact Mott activation neuron based on vanadium dioxide and its successful integration with a conductive bridge random access memory (CBRAM) crossbar array in hardware. The Mott activation neuron implements the rectified linear unit function in the analogue domain. The neuron devices consume substantially less energy and occupy two orders of magnitude smaller area than those of analogue complementary metal–oxide semiconductor implementations. The LeNet-5 network with Mott activation neurons achieves 98.38% accuracy on the MNIST dataset, close to the ideal software accuracy. We perform large-scale image edge detection using the Mott activation neurons integrated with a CBRAM crossbar array. Our findings provide a solution towards large-scale, highly parallel and energy-efficient in-memory computing systems for neural networks.

electrical and electronic engineering↗