Search NASA⌕ Search

SEARCH · Search NASA

Results for “Message passing interface (MPI)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

OpenMP in VASP: Threading and SIMD

Abstract The Vienna Ab initio Simulation Package (VASP) is a widely used electronic structure code that originally exploits process‐level parallelism through the Message Passing Interface (MPI) for work distribution within and across nodes. Architectural changes of modern parallel processors urge programmers to address thread‐ and data‐level parallelism as well to benefit most from the available compute resources within a node. We describe for VASP how to approach for an MPI + OpenMP parallelization including data‐level parallelism through OpenMP SIMD constructs together with a generic high‐level vector coding scheme. We can demonstrate an improved scalability of VASP and more than 20% gain over the MPI‐only version as well as a 2× increased performance of collective operations using the multiple‐endpoint MPI feature. The high‐level vector coding scheme applied to VASP's general gradient approximation routine gives up 9× performance gain on AVX512 platforms with the Intel compiler.

Wende, Florian↗

VeloxChem: A Python-driven density-functional theory program for spectroscopy simulations in high-performance computing environments

An open-source program named VeloxChem has been developed for the calculation of electronic real and complex linear response functions at the levels of Hartree–Fock and Kohn–Sham density functional theories. With an object-oriented program structure written in a Python/C++ layered fashion, VeloxChem enables time-efficient prototyping of novel scientific approaches without sacrificing computational efficiency, so that molecular systems involving up to and beyond 500 second-row atoms (or some 10,000 contracted and in part diffuse Gaussian basis functions) can be routinely addressed. In addition, VeloxChem is equipped with a polarizable embedding scheme for the treatment of the classical electrostatic interactions with an environment that in turn is modeled by atomic site charges and polarizabilities. The underlying hybrid message passing interface (MPI)/open multiprocessing (OpenMP) parallelization scheme makes VeloxChem suitable for execution in high-performance computing cluster environments, showing even slightly beyond linear scaling for the Fock matrix construction with use of up to 16,384 central processing unit (CPU) cores. An efficient—with respect to convergence rate and overall computational cost—multifrequency/gradient complex linear response equation solver enables calculations not only of conventional spectra, such as visible/ultraviolet/X-ray electronic absorption and circular dichroism spectra, but also time-resolved linear response signals as due to ultra-short weak laser pulses.

97 MATHEMATICS AND COMPUTING↗

Roofline model toolkit: A practical tool for architectural and program analysis

We present preliminary results of the Roofline Toolkit for multicore, many core, and accelerated architectures. This paper focuses on the processor architecture characterization engine, a collection of portable instrumented micro benchmarks implemented with Message Passing Interface (MPI), and OpenMP used to express thread-level parallelism. These benchmarks are specialized to quantify the behavior of different architectural features. Compared to previous work on performance characterization, these microbenchmarks focus on capturing the performance of each level of the memory hierarchy, along with thread-level parallelism, instruction-level parallelism and explicit SIMD parallelism, measured in the context of the compilers and run-time environments. We also measure sustained PCIe throughput with four GPU memory managed mechanisms. By combining results from the architecture characterization with the Roofline model based solely on architectural specifications, this work offers insights for performance prediction of current and future architectures and their software systems. To that end, we instrument three applications and plot their resultant performance on the corresponding Roofline model when run on a Blue Gene/Q architecture.

97 MATHEMATICS AND COMPUTING↗

DyG-DPCD: A Distributed Parallel Community Detection Algorithm for Large-Scale Dynamic Graphs

Dynamic (Temporal) graphs capture the valuable evolution of real-world systems, from the continuously evolving patterns of social interactions and genetic pathways to the dynamic fluctuations of economic forces. Detecting communities for such evolving networks poses unique challenges. Detecting and analyzing the evolution of communities within dynamic graphs unlocks valuable insights into the underlying structural and temporal patterns of real-world systems. However, the sheer volume of modern graph data and the inherent complexity of the temporal dimension pose significant challenges to scalable community detection algorithms. Addressing this gap, our work explores the limited landscape of scalable distributed-memory parallel methods specifically designed for dynamic network community detection. We propose a novel parallel algorithm, DyG-DPCD (Dynamic Graph Distributed Parallel Community Detection), to detect communities in dynamic networks using the Message Passing Interface (MPI) framework. We present a vertex-centric approach, allowing us to detect communities through local optimization. Furthermore, we enhance our baseline algorithm by incorporating three heuristics, which improve the algorithm’s performance significantly while maintaining the quality of the solutions. We demonstrate the efficiency of our algorithm by experimenting on several real-world large-scale networks with hundreds of millions of edges spanning diverse domains. Notably, DyG-DPCD achieves speedups between 25× and 30× for large networks that we experimented on using NERSC compute nodes. In conclusion, our algorithm outperforms the STINGER parallel re-agglomeration algorithm by 30×.

97 MATHEMATICS AND COMPUTING↗

Solution of the Skyrme-Hartree-Fock-Bogolyubov equations in the Cartesian deformed harmonic-oscillator basis. (VII) HFODD (v2.49t): A new version of the program

We describe the new version (v2.49t) of the code HFODD which solves the nuclear Skyrme Hartree-Fock (HF) or Skyrme Hartree-Fock-Bogolyubov (HFB) problem by using the Cartesian deformed harmonic-oscillator basis. In the new version, we have implemented the following physics features: (i) the isospin mixing and projection, (ii) the finite temperature formalism for the HFB and HF+BCS methods, (iii) the Lipkin translational energy correction method, (iv) the calculation of the shell correction. A number of specific numerical methods have also been implemented in order to deal with large-scale multi-constraint calculations and hardware limitations: (i) the two-basis method for the HFB method, (ii) the Augmented Lagrangian Method (ALM) for multi-constraint calculations, (iii) the linear constraint method based on the approximation of the RPA matrix for multi-constraint calculations, (iv) an interface with the axial and parity-conserving Skyrme-HFB code HFBTHO, (v) the mixing of the HF or HFB matrix elements instead of the HF fields. Special care has been paid to using the code on massively parallel leadership class computers. For this purpose, the following features are now available with this version: (i) the Message Passing Interface (MPI) framework, (ii) scalable input data routines, (iii) multi-threading via OpenMP pragmas, (iv) parallel diagonalization of the HFB matrix in the simplex breaking case using the ScaLAPACK library. Finally, several little significant errors of the previous published version were corrected.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A fast parallel 3D Poisson solver with longitudinal periodic and transverse open boundary conditions for space-charge simulations

A three-dimensional (3D) Poisson solver with longitudinal periodic and transverse open boundary conditions can have important applications in beam physics of particle accelerators. Here, we present a fast efficient method to solve the Poisson equation using a spectral finite-difference method. This method uses a computational domain that contains the charged particle beam only and has a computational complexity of $O(N_u (logN_{mode}))$, where $N_u$ is the total number of unknowns and $N_{mode}$ is the maximum number of longitudinal or azimuthal modes. This saves both the computational time and the memory usage of using an artificial boundary condition in a large extended computational domain. The new 3D Poisson solver is parallelized using a message passing interface (MPI) on multi-processor computers and shows a reasonable parallel performance up to hundreds of processor cores.

97 MATHEMATICS AND COMPUTING↗

A parallel and performance portable implementation of a full-field crystal plasticity model

We have developed a parallel implementation of an Elasto-Viscoplastic Fast Fourier Transform-based (EVPFFT) micromechanical solver to enable computationally efficient crystal plasticity modeling for polycrystalline materials. Our primary focus lies in achieving performance portability, allowing a single EVPFFT implementation to run optimally on various homogeneous architectures, including multi-core Central Processing Units (CPUs), as well as on heterogeneous computer architectures comprising multi-core CPUs and Graphics Processing Units (GPUs) from different vendors. To accomplish this goal, we have leveraged MATAR, a C++ software library that simplifies the creation and utilization of multidimensional dense or sparse matrix and array data structures. These data structures are designed to be portable across diverse architectures through the use of Kokkos, a performance-portable library. Additionally, we have employed the Message Passing Interface (MPI) to efficiently distribute the computational workload among processors. The heFFTe (Highly Efficient FFT for Exascale) library is used to facilitate the performance portability of the fast Fourier transforms (FFTs) computation. The computational performance of EVPFFT is evaluated and presented in terms of parallel scalability and simulation runtime on different high-performance computing (HPC) architectures. As a result, the utility of the developed framework to efficiently simulate the micro-mechanical fields in polycrystalline microstructures in engineering applications is discussed.

36 MATERIALS SCIENCE↗

Massively parallel axisymmetric fluid model for streamer discharges

A highly parallelizable fluid plasma simulation tool based upon the first-order drift-diffusion equations is discussed. Atmospheric pressure plasmas have densities and gradients that require small element sizes in order to accurately simulate the plasm resulting in computational meshes on the order of millions to tens of millions of elements for realistic size plasma reactors. To enable simulations of this nature, parallel computing is required and must be optimized for the particular problem. Here, a finite-volume, electrostatic drift-diffusion implementation for low-temperature plasma is discussed. The implementation is built upon the Message Passing Interface (MPI) library in C++ using Object Oriented Programming. The underlying numerical method is outlined in detail and benchmarked against simple streamer formation from other streamer codes. Electron densities, electric field, and propagation speeds are compared with the reference case and show good agreement. Convergence studies are also performed showing a minimal space step of approximately 4 μm required to reduce relative error to below 1% during early streamer simulation times and even finer space steps are required for longer times. Additionally, strong and weak scaling of the implementation are studied and demonstrate the excellent performance behavior of the implementation up to 100 million elements on 1024 processors. Lastly, different advection schemes are compared for the simple streamer problem to analyze the influence of numerical diffusion on the resulting quantities of interest.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

QRCODE: Massively parallelized real-time time-dependent density functional theory for periodic systems

We present a new software module, QRCODE (Quantum Research for Calculating Optically Driven Excitations), for massively parallelized real-time time-dependent density functional theory (RT-TDDFT) calculations of periodic systems in the open-source Qbox software package. Our approach utilizes a custom implementation of a fast Fourier transformation scheme that significantly reduces inter-node message passing interface (MPI) communication of the major computational kernel and shows impressive scaling up to 16,344 CPU cores. In addition to improving computational performance, QRCODE contains a suite of various time propagators for accurate RT-TDDFT calculations. As benchmark applications of QRCODE, we calculate the current density and optical absorption spectra of hexagonal boron nitride (h-BN) and photo-driven reaction dynamics of the ozone-oxygen reaction. We also calculate the second and higher harmonic generation of monolayer and multi-layer boron nitride structures as examples of large material systems. Our optimized implementation of RT-TDDFT in QRCODE enables large-scale calculations of real-time electron dynamics of chemical and material systems with enhanced computational performance and impressive scaling across several thousand CPU cores.

97 MATHEMATICS AND COMPUTING↗

Enhancing photoionization rate calculations in low-temperature plasmas using spectral methods

Photoionization plays a central role in the development of streamer discharges and other non-equilibrium plasma phenomena. It creates seed electrons, which are essential for positive streamer propagation, allowing the ionization front to move forward. Because of this, accurate modeling of photoionization is very important for predicting streamer behavior and plasma evolution. The photoionization process in air (N 2 – O 2 mixture) is often described by the Zheleznyak model (1982). This model is usually solved through Helmholtz-type equations that approximate the Zheleznyak photoionization model (Zheleznyak et al. 1982) as Partial Differential Equations (PDEs). Conventional numerical methods, such as the Finite Difference Method (FDM) or Finite Volume Method (FVM), are widely used to solve these equations. Although they are prevalent, the computational cost of these methods due to their need for matrix operations and iterative solver is demanding. To address this challenge, this work develops a spectral solver based on the Fast Fourier Transform (FFT) combined with Discrete Cosine Transform (DCT) and Discrete Sine Transform (DST) to calculate the photoionization rate efficiently in an axisymmetric cylindrical domain. This method naturally satisfies the boundary conditions used in the model and converts the PDE into algebraic ones in spectral space. Thus, avoids the need for iterative matrix solvers. When compared with FDM results, it is demonstrated that the new solver not only maintains accuracy, but also reduces the computational cost, showing a performance increase of approximately 100 compared to FDM over a wide range of problem sizes. The method is parallelized using Message Passing Interface (MPI) and has been integrated into a fluid plasma model for streamer simulation. Here, this FFT-based approach provides a fast and reliable alternative for calculating photoionization in fluid models, helping large-scale plasma simulations run faster and efficiently, and allows higher-resolution simulation without extra computational cost.

Axisymmetric system↗

Cross-scale modeling of storm surge, tide, and inundation in Mid-Atlantic Bight and New York City during Hurricane Sandy, 2012

Driven by high-resolution NAM (North America Mesoscale) and ECMWF (European Centre for Medium-Range Weather Forecasts) atmospheric fields, a 3-D unstructured grid SCHISM (Semi-implicit Cross-scale Hydroscience Integrated System Model) was applied to simulate total water level in the Mid-Atlantic Bight during Hurricane Sandy in 2012. The simulated storm surge, tide, significant wave height, and wave peak period results compared favorably with NOAA observations along US East Coast, Long Island Sound, and New York Harbor. The maximum total water level at The Battery in the New York Harbor was accurately simulated with an absolute error of less than 0.08 m (out of 2.9 m peak surge) and a timing difference less than 10 min, which includes the effects on the order of 0.1–0.3 m (5–10%) from the coastal setup induced by the surface waves. The scenario comparisons of (1) “NAM” versus “ECMWF”, (2) “2-D” versus “3-D”, and (3) “with” versus “without” wind wave model were examined. The 3-D barotropic model forced by ECMWF including the effects of wind waves performs the best, attributed to wave-induced radiation stress and reduction of bottom stress when the 3-D version is used. Simultaneously, the ELCIRC-Sub model, using a 5-m topography/bathymetry sub-grid and a regular 200-m resolution finite volume computational model grid, was developed to simulate the street-level inundation in New York City. The momentum and mass fluxes calculated by the coarser base grid model were effectively coupled with the sub-grid so that running a full-blown high-resolution base model is not required. The ELCIRC-Sub model uses MPI (Message Passing Interface) parallel computing to enlarge the coverage of the land surface and an efficient non-linear solver to improve the accuracy of the wetting-and-drying processes. The temporal comparisons of modeled water level with NOAA's tidal stations and USGS' rapid-deployed gauges showed overall performance with an average error of 0.1 m. Particularly, ELCIRC-Sub captured the profile of the highest peak surge (3.9 m), which rose 2 m within 3 hour at Kings Point, NY. The spatial comparisons of the modeled peak water level at 80 surveyed locations showed an average error less than 0.13 m. The modeled maximum inundation extent also matched well with an 80% hit rate against FEMA's Hurricane Sandy maximum flooding extent.

54 ENVIRONMENTAL SCIENCES↗

Extraction of Hydrological Proximity Measures from DEMs using Parallel Processing

Land surface topography is one of the most important terrain properties which impact hydrological, geomorphological, and ecological processes active on a landscape. In our previous efforts to develop a soil depth model based upon topographic and land cover variables, we extracted a set of hydrological proximity measures (HPMs) from a Digital Elevation Model (DEM) as potential explanatory variables for soil depth. These HPMs may also have other, more general modeling applicability in hydrology, geomorphology and ecology, and so are described here from a general perspective. The HPMs we derived are variations of the distance up to ridge points (cells with no incoming flow) and variations of the distance down to stream points (cells with a contributing area greater than a threshold), following the flow path. These HPMs were computed using the D-infinity flow model that apportions flow between adjacent neighbors based on the direction of steepest downward slope on the eight triangular facets constructed in a 3 x 3 grid cell window using the center cell and each pair of adjacent neighboring grid cells in turn. The D-infinity model typically results in multiple flow paths between 2 points on the topography, with the result that distances may be computed as the minimum, maximum or average of the individual flow paths. In addition, each of the HPMs, are calculated vertically, horizontally, and along the land surface. Previously, these HPMs were calculated using recursive serial algorithms which suffered from stack overflow problems when used to process large datasets, limiting the size of DEMs that could be analyzed using that method to approximately 7000 x 7000 cells. To overcome this limitation, we developed a message passing interface (MPI) parallel approach for calculating these HPMs. The parallel algorithms of the HPMs spatially partition the input grid into stripes which are each assigned to separate processes for computation. Each of those processes then uses a queue data structure to order the processing of cells so that each cell is visited only once and the cross-process communications that are a standard part of MPI are handled in an efficient manner. This parallel approach allows analysis of much larger DEMs as compared to the serial recursive algorithms. In this paper, we present the definitions of the HPMs, the serial and parallel algorithms used in their extraction and their potential applications in hydrology, geomorphology and ecology.

54 ENVIRONMENTAL SCIENCES↗

Scaling To A Million Cores And Beyond: Using Light-Weight Simulation to Understand The Challenges Ahead On The Road To Exascale

As supercomputers scale to 1,000 PFlop/s over the next decade, investigating the performance of parallel applications at scale on future architectures and the performance impact of different architecture choices for high-performance computing (HPC) hardware/software co-design is crucial. This paper summarizes recent efforts in designing and implementing a novel HPC hardware/software co-design toolkit. The presented Extreme-scale Simulator (xSim) permits running an HPC application in a controlled environment with millions of concurrent execution threads while observing its performance in a simulated extreme-scale HPC system using architectural models and virtual timing. This paper demonstrates the capabilities and usefulness of the xSim performance investigation toolkit, such as its scalability to 2^27 simulated Message Passing Interface (MPI) ranks on 960 real processor cores, the capability to evaluate the performance of different MPI collective communication algorithms, and the ability to evaluate the performance of a basic Monte Carlo application with different architectural parameters.

Engelmann, Christian↗

Implementation of Parallel Dynamic Simulation on Shared-Memory vs. Distributed-Memory Environments

Power system dynamic simulation computes the system response to a sequence of large disturbance, such as sudden changes in generation or load, or a network short circuit followed by protective branch switching operation. It consists of a large set of differential and algebraic equations, which is computational intensive and challenging to solve using single-processor based dynamic simulation solution. High-performance computing (HPC) based parallel computing is a very promising technology to speed up the computation and facilitate the simulation process. This paper presents two different parallel implementations of power grid dynamic simulation using Open Multi-processing (OpenMP) on shared-memory platform, and Message Passing Interface (MPI) on distributed-memory clusters, respectively. The difference of the parallel simulation algorithms and architectures of the two HPC technologies are illustrated, and their performances for running parallel dynamic simulation are compared and demonstrated.

dynamic simulation↗

Development of a CPU/GPU portable software library for Lagrangian–Eulerian simulations of liquid sprays

The Lagrangian–Eulerian method is widely used for simulations of fuel sprays in turbulent combustion because of the advantage of treating the spray droplets as discrete points. One challenge of the Lagrangian–Eulerian method is the intense computational requirement when tracking the large number of Lagrangian particles needed for high fidelity. We have developed a performance-portable library, Grit, to track the Lagrangian particles in parallel on central processing unit (CPU) and graphics processing unit (GPU) accelerated high performance computing (HPC) architectures. Grit is a C++ library which employs Message Passing Interface (MPI) for distributed memory parallelism and Kokkos programming model for on-node shared memory parallelism with performance portability across different architectures of GPUs and multi-core/manycore CPUs. The parallel algorithms, key parallel kernels, and their performances on the pre-exascale supercomputer, Summit, are presented. Grit is coupled with a direct numerical simulation (DNS) solver, S3D, for multiphase simulations. A conservative formulation has been developed and implemented in Grit for phase coupling with thermodynamic consistency. The formulation separates the conservation of mass, momentum and energy from the physical models to prevent accidental violation of conservation laws due to inconsistent models. The formulation also enforces consistent definitions of enthalpies of the fuel for both phases and the latent heat of evaporation. Finally, simulations of turbulent particle-laden flow and the evaporation of dilute turbulent spray jet are performed to verify the software implementation, and to demonstrate the scalability of Grit for large-scale multiphase simulations.

97 MATHEMATICS AND COMPUTING↗

Solution to PDEs using radial basis function finite-differences (RBF-FD) on multiple GPUs

This paper presents parallelization strategies for the radial basis function-finite difference (RBF-FD) method. As a generalized finite differencing scheme, the RBF-FD method functions without the need for underlying meshes to structure nodes. It offers high-order accuracy approximation and scales as O(N) per time step, with N being with the total number of nodes. To our knowledge, this is the first implementation of the RBF-FD method to leverage GPU accelerators for the solution of PDEs. Additionally, this implementation is the first to span both multiple CPUs and multiple GPUs. OpenCL kernels target the GPUs and inter-processor communication and synchronization is managed by the Message Passing Interface (MPI). We verify our implementation of the RBF-FD method with two hyperbolic PDEs on the sphere, and demonstrate up to 9x speedup on a commodity GPU with unoptimized kernel implementations. On a high performance cluster, the method achieves up to 7x speedup for the maximum problem size of 27,556 nodes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Scalable simulation of coupled adsorption and transport of methane in confined complex porous media with density preconditioning

The growing significance of shales and tight formations in the transition to less carbon-intensive and clean energy drives the research endeavor to understand the physics of gas flow within these systems. However, shales are composed of massively heterogeneous physical and chemical features. Most nano-sized pores connect to millimeter-scale fractures, leading to multiscale transport. These nano-scale pore throats demonstrate non-classical flow behavior, such as non-negligible slip velocities and adsorbed gas layers at the boundary. As a result, classical computational fluid dynamics models do not capture the physics. In this work, we develop a coupling scheme for the multiple-relaxation-time (MRT) lattice Boltzmann (LB) method that integrates the Peng-Robinson equation of state into a pseudo-potential interaction model to capture the physics of methane flow in irregular networks of channels that represent nano-scale porous media. We use atomistic simulations to calibrate and validate our model in slit nano-channels. We propose a preconditioning scheme to initialize the coupled transport and adsorption simulation of methane in complex porous media. The results of this implementation of LB agree with Direct Simulation Monte Carlo (DSMC) and Molecular Dynamics (MD) simulations. We then scale up the LB implementation through vectorization and indirect addressing. We parallelize it using Message Passing Interface (MPI) and OpenMP frameworks to simulate transport and adsorption in complex media with a million lattices. Additionally, we analyze the differences between coupled and transport-only simulations in two case studies and show that considering phase behavior, i.e., adsorption, can significantly change the flow behavior. This work constitutes an important step towards bridging the gap between molecular flow and system-scale behavior of complex disordered porous media.

42 ENGINEERING↗