Results for “Parallelization efficiency” — Page 17

DOE OSTI2011

Efficient data IO for a Parallel Global Cloud Resolving Model

Execution of a Global Cloud Resolving Model (GCRM) at target resolutions of 2-4 km will generate, at a minimum, 10s of Gigabytes of data per variable per snapshot. Writing this data to disk without creating a serious bottleneck in the execution of the GCRM code while also supporting efficient post-execution data analysis is a significant challenge. This paper discusses an Input/Output (IO) application programmer interface (API) for the GCRM that efficiently moves data from the model to disk while maintaining support for community standard formats, avoiding the creation of very large numbers of files, and supporting efficient analysis. Several aspects of the API will be discussed in detail. First, we discuss the output data layout which linearizes the data in a consistent way that is independent of the number of processors used to run the simulation and provides a convenient format for subsequent analyses of the data. Second, we discuss the flexible API interface that enables modelers to easily add variables to the output stream by specifying where in the GCRM code these variables are located and to flexibly configure the choice of outputs and distribution of data across files. The flexibility of the API is designed to allow model developers to add new data fields to the output as the model develops and new physics is added and also provides a mechanism for allowing users of the GCRM code itself to adjust the output frequency and the number of fields written depending on the needs of individual calculations. Third, we describe the mapping to the NetCDF data model with an emphasis on the grid description. Fourth, we describe our messaging algorithms and IO aggregation strategies that are used to achieve high bandwidth while simultaneously writing concurrently from many processors to shared files. We conclude with initial performance results.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2015

Analytic energy gradient of excited electronic state within TDDFT/MMpol framework: Benchmark tests and parallel implementation

The time-dependent density functional theory (TDDFT) has become the most popular method to calculate the electronic excitation energies, describe the excited-state properties, and perform the excited-state geometric optimization of medium and large-size molecules due to the implementation of analytic excited-state energy gradient and Hessian in many electronic structure software packages. To describe the molecules in condensed phase, one usually adopts the computationally efficient hybrid Quantum Mechanics/Molecular Mechanics (QM/MM) models. Here, we extend our previous work on the energy gradient of TDDFT/MM excited state to account for the mutual polarization effects between QM and MM regions, which is believed to hold a crucial position in the potential energy surface of molecular systems when the photoexcitation-induced charge rearrangement in the QM region is drastic. The implementation of a simple polarizable TDDFT/MM (TDDFT/MMpol) model in Q-Chem/CHARMM interface with both the linear response and the state-specific features has been realized. Several benchmark tests and preliminary applications are exhibited to confirm our implementation and assess the effects of different treatment of environmental polarization on the excited-state properties, and the efficiency of parallel implementation is demonstrated as well.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2011

Beam-beam simulation code BBSIM for particle accelerators

A highly efficient, fully parallelized, six-dimensional tracking model for simulating interactions of colliding hadron beams in high energy ring colliders and simulating schemes for mitigating their effects is described. The model uses the weak-strong approximation for calculating the head-on interactions when the test beam has lower intensity than the other beam, a look-up table for the efficient calculation of long-range beam-beam forces, and a self-consistent Poisson solver when both beams have comparable intensities. A performance test of the model in a parallel environment is presented. The code is used to calculate beam emittance and beam loss in the Tevatron at Fermilab and compared with measurements. They also present results from the studies of stwo schemes proposed to compensate the beam-beam interactions: (a) the compensation of long-range interactions in the Relativistic Heavy Ion Collider (RHIC) at Brookhaven and the Large Hadron Collider (LHC) at CERN with a current carrying wire, (b) the use of a low energy electron beam to compensate the head-on interactions in RHIC.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2013

The Data Transfer Kit: A geometric rendezvous-based tool for multiphysics data transfer

The Data Transfer Kit (DTK) is a software library designed to provide parallel data transfer services for arbitrary physics components based on the concept of geometric rendezvous. The rendezvous algorithm provides a means to geometrically correlate two geometric domains that may be arbitrarily decomposed in a parallel simulation. By repartitioning both domains such that they have the same geometric domain on each parallel process, efficient and load balanced search operations and data transfer can be performed at a desirable algorithmic time complexity with low communication overhead relative to other types of mapping algorithms. With the increased development efforts in multiphysics simulation and other multiple mesh and geometry problems, generating parallel topology maps for transferring fields and other data between geometric domains is a common operation. The algorithms used to generate parallel topology maps based on the concept of geometric rendezvous as implemented in DTK are described with an example using a conjugate heat transfer calculation and thermal coupling with a neutronics code. In addition, we provide the results of initial scaling studies performed on the Jaguar Cray XK6 system at Oak Ridge National Laboratory for a worse-case-scenario problem in terms of algorithmic complexity that shows good scaling on 0(1 x 104) cores for topology map generation and excellent scaling on 0(1 x 105) cores for the data transfer operation with meshes of O(1 x 109) elements. (authors)

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Biomolecular assembly processes in the design of novel functional materials

Nature admits innumerable examples where efficient, remarkably parallelized, and precise organization of modular building blocks (e.g., molecules, particles) leads to diverse, complex, and functional structures. Unraveling such processes in order to mimic Nature’s design strategies remains the Holy Grail in establishing a multiscale paradigm for programmable materials production. Success in this endeavor would revolutionize the efficiency and precision with which complex materials are designed and synthesized, and would spawn novel, functional structures tailored precisely for applications. Among promising approaches to this bottom-up strategy is the use of particle-tethered, complementary single-stranded DNA (ssDNA) for programming particle assembly mediated by DNA hybridization into complex structures. The overall goal of our work was to develop a comprehensive computational and theoretical strategy aimed at overcoming current barriers in self-assembly of DNA-functionalized particles (DFPs).

36 MATERIALS SCIENCE↗

DOE OSTI2016

A two-layer approach to the coupled coherent states method

In this paper, a two-layer scheme is outlined for the coupled coherent states (CCS) method, dubbed two-layer CCS (2L-CCS). The theoretical framework is motivated by that of the multiconfigurational Ehrenfest method, where different dynamical descriptions are used for different subsystems of a quantum mechanical system. This leads to a flexible representation of the wavefunction, making the method particularly suited to the study of composite systems. It was tested on a 20-dimensional asymmetric system-bath tunnelling problem, with results compared to a benchmark calculation, as well as existing CCS, matching-pursuit/split-operator Fourier transform, and configuration interaction expansion methods. The two-layer method was found to lead to improved short and long term propagation over standard CCS, alongside improved numerical efficiency and parallel scalability. These promising results provide impetus for future development of the method for on-the-fly direct dynamics calculations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2021

Parallel-In-Time Integration for Hyperbolic Problems (Final Report)

The work for the subcontract was situated in the area of parallel-in-time integration for advection-dominated and hyperbolic partial differential equations (PDEs). Parallel-in-time integration is an active area of research due to its ability to enable faster numerical simulations for applications throughout many areas of science. Over the past two decades, much progress has been made in this area; however, this progress has largely been limited to non-hyperbolic, diffusion-dominated PDEs. Given the ubiquity of numerical simulations of advection-dominated hyperbolic PDEs throughout the sciences, there is a strong need to develop efficient parallel-in-time techniques for hyperbolic problems, which was the main topic of this subcontract.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Parallel-in-Time Integration for Nonlinear Hyperbolic Problems (Final Report)

The work for the subcontract was situated in the area of parallel-in-time integration for hyperbolic partial differential equations (PDEs). Parallel-in-time integration is an active area of research due to its ability to enable faster numerical simulations for applications throughout many areas of science. Over the past two decades, much progress has been made in this area; however, this progress has largely been limited to diffusion-dominated PDEs, with some recent success in scalar linear hyperbolic PDEs. Given the ubiquity of numerical simulations of hyperbolic PDEs throughout the sciences, in particular, nonlinear hyperbolic systems, there is a strong need to develop efficient parallel-in-time techniques for hyperbolic problems beyond simple scalar and linear cases, which is the main topic of this subcontract. The main focus of the work was to further develop and perfect coarse-grid operators for the Multigrid Reduction-inTime (MGRIT) method applied to hyperbolic PDEs that were recently proposed in PhD thesis, based on a modified semi-Lagrangian approach.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Scaling and Benchmarking an Evolutionary Algorithm for Constructing Biophysical Neuronal Models

Single neuron models are fundamental for computational modeling of the brain's neuronal networks, and understanding how ion channel dynamics mediate neural function. A challenge in defining such models is determining biophysically realistic channel distributions. Here, we present an efficient, highly parallel evolutionary algorithm for developing such models, named NeuroGPU-EA. NeuroGPU-EA uses CPUs and GPUs concurrently to simulate and evaluate neuron membrane potentials with respect to multiple stimuli. We demonstrate a logarithmic cost for scaling the stimuli used in the fitting procedure. NeuroGPU-EA outperforms the typically used CPU based evolutionary algorithm by a factor of 10 on a series of scaling benchmarks. We report observed performance bottlenecks and propose mitigation strategies. Finally, we also discuss the potential of this method for efficient simulation and evaluation of electrophysiological waveforms.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Parallel algorithms for hyperdynamics and local hyperdynamics

Hyperdynamics (HD) is a method for accelerating the timescale of standard molecular dynamics (MD). It can be used for simulations of systems with an energy potential landscape that is a collection of basins, separated by barriers, where transitions between basins are infrequent. HD enables the system to escape from a basin more quickly while enabling a statistically accurate renormalization of the simulation time, thus effectively boosting the timescale of the simulation. In [Kim, Perez, Voter, J Chem Phys, 139:144110, 2013)1, a local version of HD was formulated, which exploits the intrinsic locality characteristic typical of most systems to mitigate the poor scaling properties of standard HD as the system size is increased. In this paper, we discuss how both HD and local HD can be formulated to run efficiently in parallel. We have implemented these ideas in the LAMMPS MD code, which means HD can be used with any interatomic potential LAMMPS supports. Together, these parallel methods allow simulations of any size to achieve the time acceleration offered by HD (which can be orders of magnitude), at a cost 3-5x that of standard MD. As examples, we performed two simulations of a million-atom system to model the diffusion and clustering of Pt adatoms on a large patch of Pt(100) surface for 80 and 160 μs.

74 ATOMIC AND MOLECULAR PHYSICS↗

DOE OSTI2025

Leveraging Pre-Built Catalogs and Object-Level Scheduling to Eliminate I/O Bottlenecks in HPC Environments

Modern High-Performance Computing (HPC) environments face mounting challenges due to the shift from large to small file datasets, along with an increasing number of users and parallelized applications. As HPC systems rely on Parallel File Systems (PFS), such as Lustre for data processing, performance bottlenecks stemming from Object Storage Target (OST) contention have become a significant concern. Existing solutions, such as LADS with its object-level scheduling approach, fall short in large-scale HPC environments due to their inability to effectively address metadata I/O bottlenecks and the growing number of I/O processes. This study highlights the pressing need for a comprehensive solution that tackles both OST contention and metadata I/O challenges in diverse HPC workloads. To address these challenges, we propose SwiftLoad, an object-level I/O scheduling framework that leverages a metadata catalog to enhance the performance and efficiency of parallel HPC utilities. The adoption of the metadata catalog mitigates the metadata I/O bottlenecks that commonly occur in HPC utilities, a challenge that is particularly pronounced in object-level I/O scheduling. SwiftLoad addresses OST contention and the uneven distribution of I/O processes across different OSTs through mathematical modeling and incorporates a Loader Configuration Module to regulate the number of I/O processes. Evaluated with two representative utilities—data deduplication profiling and data augmentation—SwiftLoad achieved performance improvements of up to 5.63x and 11.0x, respectively, on a production supercomputer.

HPC↗

DOE OSTI2023

UPC++ v1.0 Programmer’s Guide (Rev. 2023.9.0)

UPC++ is a C++ library that supports Partitioned Global Address Space (PGAS) programming. It is designed for writing efficient, scalable parallel programs on distributed-memory parallel computers. The key communication facilities in UPC++ are one-sided Remote Memory Access (RMA) and Remote Procedure Call (RPC). The UPC++ control model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. The PGAS memory model additionally provides one-sided RMA communication to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ also features Remote Procedure Call (RPC) communication, making it easy to move computation to operate on data that resides on remote processes. UPC++ was designed to support exascale high-performance computing, and the library interfaces and implementation are focused on maximizing scalability. In UPC++, all communication operations are syntactically explicit, which encourages programmers to consider the costs associated with communication and data movement. Moreover, all communication operations are asynchronous by default, encouraging programmers to seek opportunities for overlapping communication latencies with other useful work. UPC++ provides expressive and composable abstractions designed for efficiently managing aggressive use of asynchrony in programs. Together, these design principles are intended to enable programmers to write applications using UPC++ that perform well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

UPC++ v1.0 Programmer’s Guide (Revision 2022.3.0)

UPC++ is a C++ library that supports Partitioned Global Address Space (PGAS) programming. It is designed for writing efficient, scalable parallel programs on distributed-memory parallel computers. The key communication facilities in UPC++ are one-sided Remote Memory Access (RMA) and Remote Procedure Call (RPC). The UPC++ control model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. The PGAS memory model additionally provides one-sided RMA communication to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ also features Remote Procedure Call (RPC) communication, making it easy to move computation to operate on data that resides on remote processes. UPC++ was designed to support exascale high-performance computing, and the library interfaces and implementation are focused on maximizing scalability. In UPC++, all communication operations are syntactically explicit, which encourages programmers to consider the costs associated with communication and data movement. Moreover, all communication operations are asynchronous by default, encouraging programmers to seek opportunities for overlapping communication latencies with other useful work. UPC++ provides expressive and composable abstractions designed for efficiently managing aggressive use of asynchrony in programs. Together, these design principles are intended to enable programmers to write applications using UPC++ that perform well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

UPC++ v1.0 Programmer’s Guide, Revision 2023.3.0

UPC++ is a C++ library that supports Partitioned Global Address Space (PGAS) programming. It is designed for writing efficient, scalable parallel programs on distributed-memory parallel computers. The key communication facilities in UPC++ are one-sided Remote Memory Access (RMA) and Remote Procedure Call (RPC). The UPC++ control model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. The PGAS memory model additionally provides one-sided RMA communication to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ also features Remote Procedure Call (RPC) communication, making it easy to move computation to operate on data that resides on remote processes. UPC++ was designed to support exascale high-performance computing, and the library interfaces and implementation are focused on maximizing scalability. In UPC++, all communication operations are syntactically explicit, which encourages programmers to consider the costs associated with communication and data movement. Moreover, all communication operations are asynchronous by default, encouraging programmers to seek opportunities for overlapping communication latencies with other useful work. UPC++ provides expressive and composable abstractions designed for efficiently managing aggressive use of asynchrony in programs. Together, these design principles are intended to enable programmers to write applications using UPC++ that perform well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

UPC++ v1.0 Programmer’s Guide, Revision 2022.9.0

UPC++ is a C++ library that supports Partitioned Global Address Space (PGAS) programming. It is designed for writing efficient, scalable parallel programs on distributed-memory parallel computers. The key communication facilities in UPC++ are one-sided Remote Memory Access (RMA) and Remote Procedure Call (RPC). The UPC++ control model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. The PGAS memory model additionally provides one-sided RMA communication to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ also features Remote Procedure Call (RPC) communication, making it easy to move computation to operate on data that resides on remote processes. UPC++ was designed to support exascale high-performance computing, and the library interfaces and implementation are focused on maximizing scalability. In UPC++, all communication operations are syntactically explicit, which encourages programmers to consider the costs associated with communication and data movement. Moreover, all communication operations are asynchronous by default, encouraging programmers to seek opportunities for overlapping communication latencies with other useful work. UPC++ provides expressive and composable abstractions designed for efficiently managing aggressive use of asynchrony in programs. Together, these design principles are intended to enable programmers to write applications using UPC++ that perform well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Efficient discontinuous Galerkin implementations and preconditioners for implicit unsteady compressible flow simulations

This work presents and compares efficient implementations of high-order discontinuous Galerkin methods: a modal matrix-free discontinuous Galerkin (DG) method, a hybridizable discontinuous Galerkin (HDG) method, and a primal formulation of HDG, applied to the implicit solution of unsteady compressible flows. The matrix-free implementation allows for a reduction of the memory footprint of the solver when dealing with implicit time-accurate discretizations. HDG reduces the number of globally-coupled degrees of freedom relative to DG, at high order, by statically condensing element-interior degrees of freedom from the system in favor of face unknowns. The primal formulation further reduces the element-interior degrees of freedom by eliminating the gradient as a separate unknown. This paper introduces a p-multigrid preconditioner implementation for these discretizations and presents results for various flow problems. Benefits of the p-multigrid strategy relative to simpler, less expensive, preconditioners are observed for stiff systems, such as those arising from low-Mach number flows at high-order approximation. The p-multigrid preconditioner also shows excellent scalability for parallel computations. Finally, additional savings in both speed and memory occur with a matrix-free/reduced version of the preconditioner.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. Here, we show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

A supernodal all-pairs shortest path algorithm

We show how to exploit graph sparsity in the Floyd-Warshall algorithm for the all-pairs shortest path (Apsp) problem. Floyd-Warshall is an attractive choice for Apsp on high-performing systems due to its structural similarity to solving dense linear systems and matrix multiplication. However, if sparsity of the input graph is not properly exploited, Floyd-Warshall will perform unnecessary asymptotic work and thus may not be a suitable choice for many input graphs. To overcome this limitation, the key idea in our approach is to use the known algebraic relationship between Floyd-Warshall and Gaussian elimination, and import several algorithmic techniques from sparse Cholesky factorization, namely, fill-in reducing ordering, symbolic analysis, supernodal traversal, and elimination tree parallelism. When combined, these techniques reduce computation, improve locality and enhance parallelism. We implement these ideas in an efficient shared memory parallel prototype that is orders of magnitude faster than an efficient multi-threaded baseline Floyd-Warshall that does not exploit sparsity. Our experiments suggest that the Floyd-Warshall algorithm can compete with Dijkstra's algorithm (the algorithmic core of Johnson's algorithm) for several classes sparse graphs.

Sao, Piyush↗