Investigating Channel Pruning through Structural Redundancy Reduction - A Statistical Study
Explore the source record for details and available documents.
Li, Chengcheng↗
SEARCH · Search NASA
Results for “pruning”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The Generalization-Stability Tradeoff In Neural Network Pruning
Explore the source record for details and available documents.
Mathematics and Computing, Computer science↗
Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
Explore the source record for details and available documents.
Computer science↗
PQV-Mobile: A Combined Pruning and Quantization Toolkit to Optimize Vision Transformers for Mobile Applications
Explore the source record for details and available documents.
Computer science↗
CSB-RNN: A Faster-Than-Realtime RNN Acceleration Framework with Compressed Structured Blocks
Recurrent Neural Networks (RNN) is widely applied to temporal sequence analysis, where real-time performance is usually in demand. However, RNN suffers a heavy computational workload as the model comes with a large weight matrix. To alleviate the pain, model compression (pruning) schemes have been proposed for RNN that pruning the redundant (near-zero) weight-values. On the one hand, the non-structured pruning methods achieve a considerable pruning rate while bringing the computational irregularity, which is un-friendly to parallel-hardware. On the other hand, the existing structured pruning methods consider the hardware parallelism; However, they suffer a poor pruning rate due to the restrict constraints on pruning structure. This paper presents CSB-RNN, an optimized full-stack RNN framework with the novel compressed structured block (CSB) technique. The CSB-pruned RNN model comes with both fine-granularity that benefits the pruning rate and regular structure that facilitates the hardware-parallelism. Further, we propose a novel hardware architecture for inferencing the CSB-pruned model. Different from conventional parallel hardware, this architecture solves the block-workload imbalance issue and achieves an over 95% hardware utilization. With the experiments on 10 RNN models in 5 application domains, the CSB-RNN realizes 7×-20× lossless compression and up to 50× acceptable lossy-compression, which is 2×-7× to the prior art. With the addition of the novel hardware, the compressed-RNN inference reaches a super real-time latency of 10-400µs with FPGA implementation.
Shi, Runbin↗
Local rules for fabricating allosteric networks
Mechanical properties of disordered networks can be significantly tailored by modifying a small fraction of their bonds. This procedure has been used to design and build mechanical metamaterials with a variety of responses. A long-range “allosteric” response, where a localized input strain at one site gives rise to a localized output strain at a distant site, has been of particular interest. This work presents an approach to incorporating allosteric responses in experimental systems by pruning disordered networks in situ. Previous work has relied on computer simulations to design and predict the response of such systems using a cost function where the response of the entire network to each bond removal is used at each step to determine which bond to prune. It is not feasible to follow such a design protocol in experiments where one has access only to local response at each site. This paper presents design algorithms that allow determination of what bonds to prune based purely on the local forces in the network without employing a cost function; using only local information, allosteric networks are designed in simulations and then built out of real materials. The results show that some pruning strategies work better than others when translated into an experimental system. A method is presented to measure local stresses experimentally in disordered networks. This approach is then used to implement pruning methods to design desired responses in situ. Results from these experiments confirm that the pruning methods are robust and work in a real laboratory material.
36 MATERIALS SCIENCE↗
O3BNN-R: An Out-Of-Order Architecture for High-Performance and Regularized BNN Inference
Binarized Neural Networks (BNN) have drawn tremendous attention due to significantly reduced computational complexity and memory demand. They have especially shown great potential in cost- and power-restricted domains, such as IoT and smart edge-devices, where reaching a certain accuracy bar is often sufficient, and real-time is highly desired.In this work, we demonstrate that the highly-condensed BNN model can be shrunk significantly further by dynamically pruning irregular redundant edges. Based on two new observations on BNN-specific properties, an out-of-order (OoO) architecture – O3BNN-R, can curtail edge evaluation in cases where the binary output of a neuron can be determined early. Similar to Instruction-Level-Parallelism(ILP), these fine-grained, irregular, runtime pruning opportunities are traditionally presumed to be difficult to exploit. In order to increase the pruning opportunities, we also optimize the training process by adding 2 regularization items in the loss function (1) for pooling pruning and (2) for threshold pruning. We evaluate our design on an FPGA platform using three well-known networks, including VggNet-16, AlexNet for ImageNet, and a VGG-like network for Cifar-10.
Geng, Tong↗
Towards Efficient Convolutional Neural Networks Through Low-Error Filter Saliency Estimation
Filter saliency based channel pruning is a state-of-the-art method for deep convolutional neural network compression and acceleration. This channel pruning method ranks the importance of individual filter by estimating its impact of each filter’s removal on the training loss, and then remove the least important filters and fine-tune the remnant network. In this work, we propose a systematic channel pruning method that significantly reduces the estimation error of filter saliency. Different from existing approaches, our method largely reduces the magnitude of parameters in a network by introducing alternating direction method of multipliers (ADMM) into the pre-training procedure. Therefore, the estimation of filter saliency based on Taylor expansion is significantly improved. Extensive experiments with various benchmark network architectures and datasets demonstrate that the proposed method has a much improved unimportant filter selection capability and outperform state-of-the-art channel pruning method.
Wang, Zi↗
Mitigating Algorithmic Bias in Cancer Site Classification Models
Purpose Integrating artificial intelligence in cancer diagnostics has improved tumor classification beyond rule-based systems. Despite these advancements, these models may still encode demographic biases. We conducted a large-scale, applied bias-probing study of a deep learning–based cancer site classifier to quantify race information encoded in document embeddings. We then evaluated how performance changes when race-correlated embedding dimensions are removed in a post-training sensitivity analysis. Methods The cancer site classifier was trained using 3.5 million electronic cancer pathology reports from six of the National Cancer Institute's SEER registries. We trained a hierarchical self-attention network to generate 400-dimensional document embeddings. These embeddings were used to train two downstream, gradient-boosted decision tree classifiers: one to classify the cancer sites and another to predict racial categories. We identified overlapping features by intersecting the top 50 feature-importance rankings from the site and race models and computed their cumulative feature importance in each model. As a post hoc sensitivity analysis, we progressively pruned these overlapping dimensions, retrained the site model, and compared overall macro-F1 and accuracy, race-stratified macro-F1, and group fairness metrics on the basis of demographic parity and equalized odds before and after pruning. Results The analysis revealed minimal feature overlap between the cancer site and race prediction models, and the cumulative importance scores indicated a negligible influence of racial information on clinical predictions. Post-training pruning of overlapping features did not compromise the models' diagnostic accuracy, with a 0.07% loss in accuracy. Conclusion Our findings demonstrate that HiSAN-generated embeddings from SEER data can be used effectively in cancer site classification without significant demographic bias influencing the outcomes. Post-training pruning therefore functions as a practical audit and sensitivity check.
Shivanna, Abhishek [ORNL] (ORCID:0009000665228593)↗
Auxetic metamaterials from disordered networks
Recent theoretical work suggests that systematic pruning of disordered networks consisting of nodes connected by springs can lead to materials that exhibit a host of unusual mechanical properties. In particular, global properties such as Poisson's ratio or local responses related to deformation can be precisely altered. Tunable mechanical responses would be useful in areas ranging from impact mitigation to robotics and, more generally, for creation of metamaterials with engineered properties. However, experimental attempts to create auxetic materials based on pruningbased theoretical ideas have not been successful. In this study, we introduce a more realistic model of the networks, which incorporates angle-bending forces and the appropriate experimental boundary conditions. A sequential pruning strategy of select bonds in this model is then devised and implemented that enables engineering of specific mechanical behaviors upon deformation, both in the linear and in the nonlinear regimes. In particular, it is shown that Poisson's ratio can be tuned to arbitrary values. The model and concepts discussed here are validated by preparing physical realizations of the networks designed in this manner, which are produced by laser cutting 2D sheets and are found to behave as predicted. Furthermore, by relying on optimization algorithms, we exploit the networks' susceptibility to tuning to design networks that possess a distribution of stiffer and more compliant bonds and whose auxetic behavior is even greater than that of homogeneous networks. Finally, taken together, the findings reported here serve to establish that pruned networks represent a promising platform for the creation of unique mechanical metamaterials.
36 MATERIALS SCIENCE↗
Differential catalytic promiscuity of the alkaline phosphatase superfamily bimetallo core reveals mechanistic features underlying enzyme evolution
Members of enzyme superfamilies specialize in different reactions but often exhibit catalytic promiscuity for one another's reactions, consistent with catalytic promiscuity as an important driver in the evolution of new enzymes. Wanting to understand how catalytic promiscuity and other factors may influence evolution across a superfamily, we turned to the well-studied alkaline phosphatase (AP) superfamily, comparing three of its members, two evolutionarily distinct phosphatases and a phosphodiesterase. Here, we mutated distinguishing active-site residues to generate enzymes that had a common Zn 2+ bimetallo core but little sequence similarity and different auxiliary domains. We then tested the catalytic capabilities of these pruned enzymes with a series of substrates. A substantial rate enhancement of ~1011-fold for both phosphate mono- and diester hydrolysis by each enzyme indicated that the Zn 2+ bimetallo core is an effective mono/di-esterase generalist and that the bimetallo cores were not evolutionarily tuned to prefer their cognate reactions. In contrast, our pruned enzymes were ineffective sulfatases, and this limited promiscuity may have provided a driving force for founding the distinct one-metal-ion branch that contains all known AP superfamily sulfatases. Finally, our pruned enzymes exhibited 10 7 –10 8 -fold phosphotriesterase rate enhancements, despite absence of such enzymes within the AP superfamily. We speculate that the superfamily active-site architecture involved in nucleophile positioning prevents accommodation of the additional triester substituent. Overall, we suggest that catalytic promiscuity, and the ease or difficulty of remodeling and building onto existing protein scaffolds, have greatly influenced the course of enzyme evolution. Uncovering principles and properties of enzyme function, promiscuity, and repurposing provides lessons for engineering new enzymes.
59 BASIC BIOLOGICAL SCIENCES↗
Unsupervised atomic data mining via multi-kernel graph autoencoders for machine learning force fields
Constructing a chemically diverse dataset while avoiding sampling bias is critical to training efficient and generalizable force fields. However, in computational chemistry and materials science, many common dataset generation techniques are prone to oversampling regions of the potential energy surface. Furthermore, these regions can be difficult to identify and isolate from each other or may not align well with human intuition, making it challenging to systematically remove bias in the dataset. While traditional clustering and pruning (down-sampling) approaches can be useful for this, they can often lead to information loss or a failure to properly identify distinct regions of the potential energy surface due to difficulties associated with the high dimensionality of atomic descriptors. In this work, we introduce the Multi-kernel Edge Attention-based Graph Autoencoder (MEAGraph) model, an unsupervised approach for analyzing atomic datasets. MEAGraph combines multiple linear kernel transformations with attention-based message passing to capture geometric sensitivity and enable effective dataset pruning without relying on labels or extensive training. Demonstrated applications on niobium, tantalum, and iron datasets show that MEAGraph efficiently groups similar atomic environments, allowing for the use of basic pruning techniques for removing sampling bias. This approach provides an effective method for representation learning and clustering that can be used for data analysis, outlier detection, and dataset optimization.
Materials science↗
Binary Complex Neural Network Acceleration on FPGA
Being able to learn from complex data with phase information is imperative for many signal processing applications. Today’s real-valued deep neural networks (DNNs) have shown efficiency in latent information analysis but fall short when applied to the complex domain. Deep complex networks (DCN) , in contrast, can learn from complex data, but have high computational costs; therefore, they cannot satisfy the instant decision making requirements of many deployable systems dealing with short observations or short signal bursts. Recent, Binarized Complex Neural Network (BCNN), which integrates DCNs with binarized neural networks (BNN), shows great potential in classifying complex data in real-time. In this paper, we propose a structural pruning based accelerator of BCNN, which is able to provide more than 5000 frames/s inference throughput on edge devices. The high performance comes from both the algorithm and hardware sides. On the algorithm side, we conduct structural pruning to the original BCNN models and obtain 20 × pruning rates with negligible accuracy loss; on the hardware side, we propose a novel 2D convolution operation accelerator for the binary complex neural network. Experimental results show that the proposed design works with over 90% utilization and is able to achieve the inference throughput of 5882 frames/s and 4938 frames/s for complex NIN-Net and ResNet-18 using CIFAR-10 dataset and Alveo U280 Board.
Peng, Hongwu↗
Standard grids for high-precision integration of modern density functionals: SG-2 and SG-3
Density-functional approximations developed in the past decade necessitate the use of quadrature grids that are far more dense than those required to integrate older generations of functionals. This category of difficult-to-integrate functionals includes meta-generalized gradient approximations, which depend on orbital gradients and/or the Laplacian of the density, as well as functionals based on B97 and the popular “Minnesota” class of functionals, each of which contain complicated and/or oscillatory expressions for the exchange inhomogeneity factor. Following a strategy introduced previously by Gill and co-workers to develop the relatively sparse “SG-0” and “SG-1” standard quadrature grids, we introduce two higher-quality grids that we designate SG-2 and SG-3, obtained by systematically “pruning” medium- and high-quality atom-centered grids. The pruning procedure affords computational speedups approaching a factor of two for hybrid functionals applied to systems of ~100 atoms, without significant loss of accuracy. The grid dependence of several popular density functionals is characterized for various properties.
74 ATOMIC AND MOLECULAR PHYSICS↗
Uncontrolled Learning: Codesign of Neuromorphic Hardware Topology for Neuromorphic Algorithms
Neuromorphic computing has the potential to revolutionize future technologies and our understanding of intelligence, yet it remains challenging to realize in practice. The learning-from-mistakes algorithm, inspired by the brain's simple learning rules of inhibition and pruning, is one of the few brain-like training methods. This algorithm is implemented in neuromorphic memristive hardware through a codesign process that evaluates essential hardware trade-offs. While the algorithm effectively trains small networks as binary classifiers and perceptrons, performance declines significantly with increasing network size unless the hardware is tailored to the algorithm. This work investigates the trade-offs between depth, controllability, and capacity—the number of learnable patterns—in neuromorphic hardware. This highlights the importance of topology and governing equations, providing theoretical tools to evaluate a device's computational capacity based on its measurements and circuit structure. The findings show that breaking neural network symmetry enhances both controllability and capacity. Additionally, by pruning the circuit, neuromorphic algorithms in all-memristive circuits can utilize stochastic resources to create local contrasts in network weights. Through combined experimental and simulation efforts, the parameters are identified that enable networks to exhibit emergent intelligence from simple rules, advancing the potential of neuromorphic computing.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Robust system size reduction of discrete fracture networks: a multi-fidelity method that preserves transport characteristics
In this paper, we propose a multi-fidelity system reduction technique that uses weighted graphs paired with three-dimensional discrete fracture network (DFN) modelling for efficient simulation of subsurface flow and transport in fractured media. DFN models are used to simulate flow and transport in subsurface fractured rock with low-permeability. One method to alleviate the heavy computational overhead associated with these simulations is to reduce the size of the DFN using a graph representation of it to identify the primary flow sub-network and only simulate flow and transport thereon. The first of these methods used unweighted graphs constructed solely on DFN topology and could be used for accurate predictions of first-passage times. However, these techniques perform poorly when predicting later stages of the mass breakthrough. We utilize a weighted-graph representation of the DFN where edge weights are based on hydrological parameters in the DFN that allows us to exploit the kinematic quantities derivable a posteriori from the flow solution obtained on the graph representation of the DFN to perform system reduction and predict the later stages of the breakthrough curve with high fidelity. We also propose and demonstrate the use of an adaptive pruning algorithm with error control that produces a pruned DFN sub-network whose predicted mass breakthrough agrees with the original DFN within a user-specified tolerance. Lastly, the method allows for the level of accuracy to be a user-controlled parameter.
58 GEOSCIENCES↗
Polishing the Gold Standard: The Role of Orbital Choice in CCSD(T) Vibrational Frequency Prediction
While CCSD(T) with spin-restricted Hartree-Fock (RHF) orbitals has long been lauded for its ability to accurately describe closed-shell interactions, the performance of CCSD(T) on open-shell species is much more erratic, especially when using a spin-unrestricted HF (UHF) reference. Previous studies have shown improved treatment of open-shell systems when a non-HF set of molecular orbitals, like Brueckner or Kohn-Sham density functional theory (DFT) orbitals, is used as a reference. Inspired by the success of regularized orbital-optimized second-order Møller-Plesset perturbation theory (κ-OOMP2) orbitals as reference orbitals for MP3, we investigate the use of κ-OOMP2 orbitals and various DFT orbitals as reference orbitals for CCSD(T) calculations of the corrected ground-state harmonic vibrational frequencies of a set of 36 closed-shell (29 neutrals, 6 cations, 1 anion) and 59 open-shell diatomic species (38 neutrals, 15 cations, 6 anions). The aug-cc-pwCVTZ basis set is used for all calculations. The use of κ-OOMP2 orbitals in this context alleviates difficult cases observed for both UHF orbitals and OOMP2 orbitals. Removing two multireference systems and 12 systems with ambiguous experimental data leaves a pruned data set. Overall performance on the pruned data set highlights CCSD(T) with a B97 orbital reference (CCSD(T):B97), CCSD(T) with a κ-OOMP2 orbital reference (CCSD(T):κ-OOMP2), and CCSD(T) with a B97M-rV orbital reference (CCSD(T):B97M-rV) with RMSDs of 8.48 cm -1 , and 8.50 cm -1 , and 8.75 cm -1 respectively, outperforming CCSD(T):UHF by nearly a factor of 5. Moreover, the performance on the closed- and open-shell subsets shows these methods are able to treat open-shell and closed-shell systems with comparable accuracy and robustness. CCSD(T) with RHF orbitals is seen to improve upon UHF for the closed-shell species, while spatial symmetry breaking in a number of restricted open-shell HF (ROHF) references leads CCSD(T) with ROHF reference orbitals to exhibit the poorest statistical performance of all methods surveyed for open-shell species. The use of κ-OOMP2 orbitals has also proven useful in diagnosing multireference character that can hinder the reliability of CCSD(T).
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Isochronic development of cortical synapses in primates and mice
Abstract The neotenous, or delayed, development of primate neurons, particularly human ones, is thought to underlie primate-specific abilities like cognition. We tested whether synaptic development follows suit—would synapses, in absolute time, develop slower in longer-lived, highly cognitive species like non-human primates than in shorter-lived species with less human-like cognitive abilities, e.g., the mouse? Instead, we find that excitatory and inhibitory synapses in the maleMus musculus(mouse) andRhesus macaque(primate) cortex form at similar rates, at similar times after birth. Primate excitatory and inhibitory synapses and mouse excitatory synapses also prune in such an isochronic fashion. Mouse inhibitory synapses are the lone exception, which are not pruned and instead continuously added throughout life. The monotony of synaptic development clocks across species with disparate lifespans, experiences, and cognitive abilities argues that such programs are likely orchestrated by genetic events rather than experience.
Science & Technology - Other Topics↗