Commutative Data Reordering: A New Technique to Reduce Data Movement Energy on Sparse Inference Workloads.
Abstract not provided.
Feinberg, Benjamin↗
SEARCH · Search NASA
Results for “reduced data movement”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Commutative Data Reordering: A New Technique to Reduce Data Movement Energy on Sparse Inference Workloads.
Abstract not provided.
Feinberg, Benjamin↗
Milestone M7 Report: Reducing Excess Data Movement Part 2
This milestone evaluates techniques to measure and, if possible, reduce data movement across all levels of the memory hierarchy, focusing on CPU/GPU page level data movement and on intra-GPU memory hierarchy. We quantitatively evaluate the efficacy of the techniques in reducing data movement and measure how performance tracks data movement reduction. We study a small collection of benchmarks and proxy mini-apps that run on advanced pre-exascale GPUs and on the Accelsim GPU simulator. Our approach has two thrusts: to measure advanced data movement reduction directives and techniques on the newest available GPUs, and to evaluate our benchmark set on simulated GPUs configured with architectural refinements to reduce data movement. We primarily evaluated NVidia-based architectures due to the unavailability of AMD GPU hardware and tools until very recently.
97 MATHEMATICS AND COMPUTING↗
Milestone M6 Report: Reducing Excess Data Movement Part 1
This is the second in a sequence of three Hardware Evaluation milestones that provide insight into the following questions: What are the sources of excess data movement across all levels of the memory hierarchy, going out to the network fabric? What can be done at various levels of the hardware/software hierarchy to reduce excess data movement? How does reduced data movement track application performance? The results of this study can be used to suggest where the DOE supercomputing facilities, working with their hardware vendors, can optimize aspects of the system to reduce excess data movement. Quantitative analysis will also benefit systems software and applications to optimize caching and data layout strategies. Another potential avenue is to answer cost-benefit questions, such as those involving memory capacity versus latency and bandwidth. This milestone focuses on techniques to reduce data movement, quantitatively evaluates the efficacy of the techniques in accomplishing that goal, and measures how performance tracks data movement reduction. We study a small collection of benchmarks and proxy mini-apps that run on pre-exascale GPUs and on the Accelsim GPU simulator. Our approach has two thrusts: to measure advanced data movement reduction directives and techniques on the newest available GPUs, and to evaluate our benchmark set on simulated GPUs configured with architectural refinements to reduce data movement.
97 MATHEMATICS AND COMPUTING↗
DEGAS: Dynamic Exascale Global Address Space Programming Environments
The Dynamic, Exascale Global Address Space programming environment (DEGAS) project will develop the next generation of programming models and runtime systems to meet the challenges of Exascale computing. The Berkeley part of the project concentrated on communication-optimal code generation to optimize speed and energy efficiency by reducing data movement. Our work developed communication lower bounds, and/or communication avoiding algorithms (that either meet the lower bound, or do much less communication than their conventional counterparts) for a variety of algorithms, including linear algebra, machine learning and genomics. The Berkeley part of the project concentrated on communication-optimal code generation to optimize speed and energy efficiency by reducing data movement. Our work developed communication lower bounds, and/or communication avoiding algorithms (that either meet the lower bound, or do much less communication than their conventional counterparts) for a variety of algorithms, including linear algebra, machine learning and genomics.
97 MATHEMATICS AND COMPUTING↗
Performance analysis of distributed symmetric sparse matrix vector multiplication algorithm for multi-core architectures
In this article, sparse matrix vector multiply (SpMVM) is an important kernel that frequently arises in high performance computing applications. Due to its low arithmetic intensity, several approaches have been proposed in literature to improve its scalability and efficiency in large scale computations. In this paper, our target systems are high end multi-core architectures and we use messaging passing interface + open multiprocessing hybrid programming model for parallelism. We analyze the performance of recently proposed implementation of the distributed symmetric SpMVM, originally developed for large sparse symmetric matrices arising in ab initio nuclear structure calculations. We also study important features of this implementation and compare with previously reported implementations that do not exploit underlying symmetry. Our SpMVM implementations leverage the hybrid paradigm to efficiently overlap expensive communications with computations. Our main comparison criterion is the "CPU core hours" metric, which is the main measure of resource usage on supercomputers. We analyze the effects of topology-aware mapping heuristic using simplified network load model. Furthermore, we have tested the different SpMVM implementations on two large clusters with 3D Torus and Dragonfly topology. Our results show that the distributed SpMVM implementation that exploits matrix symmetry and hides communication yields the best value for the "CPU core hours" metric and significantly reduces data movement overheads.
97 MATHEMATICS AND COMPUTING↗
ZFP Hardware Implementation
ZFP is a floating-point compression format gaining traction in high-performance computing applications. A software implementation has demonstrated the ability to reduce data movement across communication channels and to reduce the footprint of floating-point arrays in memory. Nevertheless, any benefit in performance is limited to the spare compute cycles available before reaching bandwidth limitations. A hardware implementation of ZFP has the potential to substantially improve HPC application performance. Hardware compression speedup over software alone on a single CPU core ranges from 15x for 1-D arrays to over 200x for 3-D arrays. The implementation of ZFP is sourced in SystemC to facilitate its evaluation in various architectures. The encode pipeline consists of several modules. Uncompressed blocks of floating-point numbers in IEEE format flow into the pipeline in a stream and a compressed bitstream flows out from the pipeline. The decode pipeline is the inverse of the encode pipeline. A modular design enables formats other than IEEE (such as posits) to be considered with minor adaptations. Best performance will be realized from the hardware ZFP unit when batches of blocks are processed at a time. The implementation supports 1-D, 2- D and 3-D blocks of floating-point numbers. The hardware implementation of ZFP has been validated with the software implementation of ZFP. C++ template parameters are used to specify the bit width of floating-point numbers and the array dimension of the encoder. A test bench program has been created with several test cases, some with continuous data that is ideal for ZFP, and others with extreme cases containing numbers near or at the maximum or minimum values supported by the number format.
Lloyd, GS↗
Parthenon—a performance portable block-structured adaptive mesh refinement framework
On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multidimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of 1.7 × 10 13 zone-cycles/s on 9216 nodes (73,728 logical GPUs) at [Formula: see text] weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.
97 MATHEMATICS AND COMPUTING↗
ZFP Hardware Implementation
As core counts increase in new HPC systems with comparatively little increase in memory bandwidth, the trend is an effective decrease in memory bandwidth per core. Other bandwidth limitations in HPC systems exist between CPU and GPU memory, between system nodes, and between node memory and storage. Compression of floating-point data has the potential to reduce data movement and pressure across these communication channels. Furthermore, it has the potential to reduce the footprint of floating-point arrays stored in memory. ZFP, implemented in software, is gaining traction as an effective method in floating-point compression; however, performance gains are limited to the spare compute cycles available before reaching the bandwidth limitations of the communication channel. A hardware implementation of ZFP has the potential to raise the bar on performance. From the inception of ZFP, it was designed to accommodate a hardware implementation.
42 ENGINEERING↗
Arrangements for communicating data in a computing system using multiple processors
Systems and methods for reducing data movement in a computer system. The systems and methods use information or knowledge about the structure of an algorithm, operations to be executed at a receiving processing unit, variables or subsets or groups of variables in a distributed algorithm, or other forms of contextual information, for reducing the number of bits transmitted from at least one transmitting processing unit to at least one receiving processing unit or storage device.
Gonzalez, Juan Guillermo↗
Arrangements for communicating and processing data in a computing system
Systems and methods for reducing data movement in a computer system. The systems and methods use information or knowledge about the structure of an algorithm, operations to be executed at a receiving processing unit, variables or subsets or groups of variables in a distributed algorithm, or other forms of contextual information, for reducing the number of bits transmitted from at least one transmitting processing unit to at least one receiving processing unit or storage device.
Gonzalez, Juan Guillermo↗
Lossy checkpoint compression in full waveform inversion: a case study with ZFPv0.5.5 and the overthrust model
This paper proposes a new method that combines checkpointing methods with error-controlled lossy compression for large-scale high-performance full-waveform inversion (FWI), an inverse problem commonly used in geophysical exploration. This combination can significantly reduce data movement, allowing a reduction in run time as well as peak memory. In the exascale computing era, frequent data transfer (e.g., memory bandwidth, PCIe bandwidth for GPUs, or network) is the performance bottleneck rather than the peak FLOPS of the processing unit. Like many other adjoint-based optimization problems, FWI is costly in terms of the number of floating-point operations, large memory footprint during backpropagation, and data transfer overheads. Past work for adjoint methods has developed checkpointing methods that reduce the peak memory requirements during backpropagation at the cost of additional floating-point computations. Combining this traditional checkpointing with error-controlled lossy compression, we explore the three-way tradeoff between memory, precision, and time to solution. We investigate how approximation errors introduced by lossy compression of the forward solution impact the objective function gradient and final inverted solution. Empirical results from these numerical experiments indicate that high lossy-compression rates (compression factors ranging up to 100) have a relatively minor impact on convergence rates and the quality of the final solution.
58 GEOSCIENCES↗
Investigating Inherent Numerical Stabilization for the Moist, Compressible, Non‐Hydrostatic Euler Equations on Collocated Grids
This study investigates inherent numerical dissipation due to upwind fluxes and reconstruction strategies for collocated Finite-Volume integration of the Euler equations. Idealized supercell simulations are used without any explicit dissipation. Flux terms are split into: mass flux, pressure, and advected quantities. They are computed with the following upwind strategies: central, advectively upwind, and acoustically upwind. This is performed for third and ninth-order-accurate reconstructions with and without Weighted Essentially Non-Oscillatory limiting. Acoustic-only upwinding for pressure and mass flux terms and advective-only upwinding for advected quantities is the most flexible simplification found. It reduces data movement and computations. Assuming a constant speed of sound in acoustic upwinding gives similar results to using the true speed of sound. Dissipation from upwind adapts automatically to grid spacing, time step, reconstruction accuracy, and flow smoothness. While stability is maintained even at 21st-order spatial accuracy, there is a limit to the spatial order of accuracy for which upwinding alone can create a realizable solution in the conditions of this study. Convex combinations of upwind and central solutions for flux terms also reduced dissipation, but as the central proportion grows, solutions become physically unrealizable. The range of length scales of the kinetic energy spectra can be extended along k −5/3 to smaller spatial scales by reducing dissipation either with higher-order reconstructions or using convex combinations of upwind and central fluxes. However, not all extensions of the length scale range along k −5/3 exhibit physically realizable solutions, even though the spectra appear to be physical.
54 ENVIRONMENTAL SCIENCES↗
Compute in‐Memory with Non‐Volatile Elements for Neural Networks: A Review from a Co‐Design Perspective
Abstract Deep learning has become ubiquitous, touching daily lives across the globe. Today, traditional computer architectures are stressed to their limits in efficiently executing the growing complexity of data and models. Compute‐in‐memory (CIM) can potentially play an important role in developing efficient hardware solutions that reduce data movement from compute‐unit to memory, known as the von Neumann bottleneck. At its heart is a cross‐bar architecture with nodal non‐volatile‐memory elements that performs an analog multiply‐and‐accumulate operation, enabling the matrix‐vector‐multiplications repeatedly used in all neural network workloads. The memory materials can significantly influence final system‐level characteristics and chip performance, including speed, power, and classification accuracy. With an over‐arching co‐design viewpoint, this review assesses the use of cross‐bar based CIM for neural networks, connecting the material properties and the associated design constraints and demands to application, architecture, and performance. Both digital and analog memory are considered, assessing the status for training and inference, and providing metrics for the collective set of properties non‐volatile memory materials will need to demonstrate for a successful CIM technology.
36 MATERIALS SCIENCE↗
Scalable line and plane relaxation in a parallel structured multigrid solver
The efficient solution of sparse, linear systems that arise through the discretization of partial differential equations remains a key challenge for a range of high performance scientific simulations. One approach for reducing data movement and improving performance is by exposing and exploiting structure in a problem through the use of robust structured multilevel solvers. By choosing coarsening that preserves the structure of the problem, these methods maintain efficient structured computation and communication throughout the multigrid hierarchy. However, when coarsening is not permitted to be dependent on the operator, anisotropy must be addressed by the smoother — producing error compatible for coarse-grid correction with structured coarsening. Here, the components required in a scalable parallel structured solver are described with a focus on memory and communication efficiency of robust smoothers. While the implementation of communication and memory reduction techniques in smoothers integrated in a complete 3D solver present a significant engineering challenge, a novel approach is proposed that addresses these challenges systematically through a change to the solver’s execution model. Enabled by user-level threading paired with a set of data and communication abstractions, this approach permits seamless aggregation of communication in plane smoothers — directly reusing code for a 2D distributed multilevel cycle. Results show an effective reduction in communication costs for coarse-grid problems, and result in a speedup of 8.7x in smoothing routines shown in Fig. 12 using this approach. This produces a significant improvement to strong scalability while maintaining favorable weak scaling behavior. Finally, a parallel scaling study using a series of refined meshes is included that demonstrates the effectiveness of this approach in an application of interest.
97 MATHEMATICS AND COMPUTING↗
Porting hypre to heterogeneous computer architectures: Strategies and experiences
We report that linear systems are occurring in many applications, and solving them can take a large amount of the total simulation time. The high performance library hypre provides a variety of interfaces and linear solvers, including various multigrid methods, that have achieved good scalability on a variety of homogeneous parallel computer architectures. Heterogeneous architectures with nodes that have both CPUs and accelerators provide new challenges, since they require more fine-grained parallelism and reduced data movement between different memories on a single node as well as across nodes. We will discuss our experiences and strategies to port hypre to heterogeneous computers with accelerators, including the design of a new memory model, the use of abstractions, the BoxLoop macros in the structured and semi-structured interfaces, and the restructuring of algebraic multigrid (AMG) into modular components. We present numerical experiments comparing CPU and GPU performance for several test problems.
97 MATHEMATICS AND COMPUTING↗
A Positive-Definite, WENO-Limited, High-Order Finite Volume Solver for 2-D Transport on the Cubed Sphere Using an ADER Time Discretization
Modern computer architectures reward added computation if it reduces algorithmic dependence, reduces data movement, increases accuracy/robustness, and improves memory accesses. The driving motive for this study is to develop a numerical algorithm that respects these constraints while improving accuracy and robustness. This study introduces the ADER-DT (Arbitrary DERivatives in time and space-differential transform) time discretization to positive-definite, weighted essentially nonoscillatory (WENO)-limited, finite volume transport on the cubed sphere in lieu of semidiscrete integrators. The cost of the ADER-DT algorithm is significantly improved from previous implementations without affecting accuracy. A new function-based WENO implementation is also detailed for use with the ADER-DT time discretization. While ADER-DT costs about 1.5 times more than a fourth-order, five-stage strong stability preserving Runge-Kutta (SSPRK4) method, it is far more computationally dense (which is advantageous on accelerators such as graphics processing units), and it has a larger effective maximum stable time step. ADER-DT errors converge more quickly with grid refinement than SSPRK4, giving 6.5 times less error in the L ∞ norm than SSPRK4 at the highest refinement level for smooth data. For nonsmooth data, ADER-DT resolves C 0 discontinuities more sharply. For a complex flow field, ADER exhibits less phase error than SSPRK4. In conclusion, improving both accuracy and robustness as well as better respecting modern computational efficiency requirements, we believe the method presented herein is competitive for efficiently transporting tracers over the sphere for applications targeting modern computing architectures.
97 MATHEMATICS AND COMPUTING↗
SharP Data Constructs: Data Constructs to Enable Data-Centric Computing
Extreme-scale applications (i.e., Big-Compute) are becoming increasingly data-intensive, i.e., producing and consuming increasingly large amounts of data. The HPC systems traditionally used for these applications are now used for Big-Data applications such as data analytics, social network analysis, machine learning, and genomics. As a consequence of these trends, the system architecture should be flexible and data-centric. This can already be witnessed in the pre-exascale systems with TBs of on-node hierarchical and heterogeneous memories, PBs of system memory, low-latency, high-throughput networks, and many threaded cores. As such, the pre-exascale systems suit the needs of both Big-Compute and Big-Data applications. Though the system architecture is flexible enough to support both Big-Compute and Big-Data, we argue there is a software gap. Particularly, we need data-centric abstractions to leverage the full potential of the system, i.e., there is a need for native support for data resilience, the ability to express data locality and affinity, mechanisms to reduce data movement, the ability to share data, and abstractions to express User's data usage and data access patterns. In this paper, we (i) show the need for taking a holistic approach towards data-centric abstractions, (ii) show how these approaches were realized in the SHARed data-structure centric Programming abstraction (SharP) library, a data-structure centric programming abstraction, and (iii) apply these approaches to a variety of applications that demonstrate its usefulness. Particularly, we apply these approaches to QMCPack and the Graph500 benchmark and demonstrate the advantages of this approach on extreme-scale systems.
Aderholdt, William Ferrol↗