Search NASA⌕ Search

SEARCH · Search NASA

Results for “Distributed memory”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Development of a CPU/GPU portable software library for Lagrangian–Eulerian simulations of liquid sprays

The Lagrangian–Eulerian method is widely used for simulations of fuel sprays in turbulent combustion because of the advantage of treating the spray droplets as discrete points. One challenge of the Lagrangian–Eulerian method is the intense computational requirement when tracking the large number of Lagrangian particles needed for high fidelity. We have developed a performance-portable library, Grit, to track the Lagrangian particles in parallel on central processing unit (CPU) and graphics processing unit (GPU) accelerated high performance computing (HPC) architectures. Grit is a C++ library which employs Message Passing Interface (MPI) for distributed memory parallelism and Kokkos programming model for on-node shared memory parallelism with performance portability across different architectures of GPUs and multi-core/manycore CPUs. The parallel algorithms, key parallel kernels, and their performances on the pre-exascale supercomputer, Summit, are presented. Grit is coupled with a direct numerical simulation (DNS) solver, S3D, for multiphase simulations. A conservative formulation has been developed and implemented in Grit for phase coupling with thermodynamic consistency. The formulation separates the conservation of mass, momentum and energy from the physical models to prevent accidental violation of conservation laws due to inconsistent models. The formulation also enforces consistent definitions of enthalpies of the fuel for both phases and the latent heat of evaporation. Finally, simulations of turbulent particle-laden flow and the evaporation of dilute turbulent spray jet are performed to verify the software implementation, and to demonstrate the scalability of Grit for large-scale multiphase simulations.

97 MATHEMATICS AND COMPUTING↗

Advanced operator splitting-based semi-implicit spectral method to solve the binary phase-field crystal equations with variable coefficients

We present an efficient method to solve numerically the equations of dissipative dynamics of the binary phase-field crystal model proposed by Elder et al. [K.R. Elder, M. Katakowski, M. Haataja, M. Grant, Phys. Rev. B 75 (2007) 064107] characterized by variable coefficients. Using the operator splitting method, the problem has been decomposed into sub-problems that can be solved more efficiently. A combination of non-trivial splitting with spectral semi-implicit solution leads to sets of algebraic equations of diagonal matrix form. Extensive testing of the method has been carried out to find the optimum balance among errors associated with time integration, spatial discretization, and splitting. We show that our method speeds up the computations by orders of magnitude relative to the conventional explicit finite difference scheme, while the costs of the pointwise implicit solution per timestep remains low. Also we show that due to its numerical dissipation, finite differencing can not compete with spectral differencing in terms of accuracy. In addition, we demonstrate that our method can efficiently be parallelized for distributed memory systems, where an excellent scalability with the number of CPUs is observed.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A composite grid solver for conjugate heat transfer in fluid-structure systems

We describe a numerical method for modeling temperature-dependent fluid flow coupled to heat transfer in solids. This approach to conjugate heat transfer can be used to compute transient and steady state solutions to a wide range of fluid-solid systems in complex two- and three-dimensional geometry. Fluids are modeled with the temperature-dependent incompressible Navier-Stokes equations using the Boussinesq approximation. Solids with heat transfer are modeled with the heat equation. Appropriate interface equations are applied to couple the solutions across different domains. The computational region is divided into a number of sub-domains corresponding to fluid domains and solid domains. There may be multiple fluid domains and multiple solid domains. Each fluid or solid sub-domain is discretized with an overlapping grid. The entire region is associated with a composite grid which is the union of the overlapping grids for the sub-domains. A different physics solver (fluid solver or solid solver) is associated with each sub-domain. A higher-level multi-domain solver manages the entire solution process. We propose and analyze some centered discrete approximations to the interface equations that have some desirable stability properties. The coupled interface equations may be solved directly when using explicit time-stepping methods in the sub-domains, resulting in a strongly coupled approach. The stability of the interface treatment in this case is independent of the relative sizes of the material properties in the two domains with the time-step only depending on the usual von Neumann conditions for each sub-domain. For implicit time-stepping methods we solve the interface equations in a weakly coupled fashion to avoid forming a coupled implicit system across all sub-domains. The convergence of this approach does depend on the relative sizes of the thermal conductivities and diffusivities. We analyze different iteration strategies for solving these implicit equations including the use of mixed (Robin) approximations at the interface. Numerical results are presented to illustrate the method. The accuracy of the technique is verified using the method of analytic solutions and by computing the solution to some heat exchanger problems where the exact solution is known. The technique is also applied to the modeling of an inertial-confinement-fusion hohlraum target and the flow of coolant past an hexagonal array of heated fuel rods. The multi-domain solver runs in parallel on distributed memory computers and some parallel results are provided.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

On the use of flux limiters in the discrete ordinates method for 3D radiation calculations in absorbing and scattering media

The application of flux limiters to the discrete ordinates method (DOM), S{sub N}, for radiative transfer calculations is discussed and analyzed for 3D enclosures for cases in which the intensities are strongly coupled to each other such as: radiative equilibrium and scattering media. A Newton-Krylov iterative method (GMRES) solves the final systems of linear equations along with a domain decomposition strategy for parallel computation using message passing libraries in a distributed memory system. Ray effects due to angular discretization and errors due to domain decomposition are minimized until small variations are introduced by these effects in order to focus on the influence of flux limiters on errors due to spatial discretization, known as numerical diffusion, smearing or false scattering. Results are presented for the DOM-integrated quantities such as heat flux, irradiation and emission. A variety of flux limiters are compared to 'exact' solutions available in the literature, such as the integral solution of the RTE for pure absorbing-emitting media and isotropic scattering cases and a Monte Carlo solution for a forward scattering case. Additionally, a non-homogeneous 3D enclosure is included to extend the use of flux limiters to more practical cases. The overall balance of convergence, accuracy, speed and stability using flux limiters is shown to be superior compared to step schemes for any test case.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A fast parallel Poisson solver on irregular domains applied to beam dynamics simulations

We discuss the scalable parallel solution of the Poisson equation within a Particle-In-Cell (PIC) code for the simulation of electron beams in particle accelerators of irregular shape. The problem is discretized by Finite Differences. Depending on the treatment of the Dirichlet boundary the resulting system of equations is symmetric or 'mildly' nonsymmetric positive definite. In all cases, the system is solved by the preconditioned conjugate gradient algorithm with smoothed aggregation (SA) based algebraic multigrid (AMG) preconditioning. We investigate variants of the implementation of SA-AMG that lead to considerable improvements in the execution times. We demonstrate good scalability of the solver on distributed memory parallel processor with up to 2048 processors. We also compare our iterative solver with an FFT-based solver that is more commonly used for applications in beam dynamics.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Direct pore-level modeling of incompressible fluid flow in porous media

We present a dynamic particle-based model for direct pore-level modeling of incompressible viscous fluid flow in disordered porous media. The model is capable of simulating flow directly in three-dimensional high-resolution micro-CT images of rock samples. It is based on moving particle semi-implicit (MPS) method. We modify this technique in order to improve its stability for flow in porous media problems. Using the micro-CT image of a rock sample, the entire medium, i.e., solid and fluid, is discretized into particles. The incompressible Navier-Stokes equations are then solved for each particle using the MPS summations. The model handles highly irregular fluid-solid boundaries effectively. An algorithm to split and merge fluid particles is also introduced. To handle the computational load, we present a parallel version of the model that runs on distributed memory computer clusters. The accuracy of the model is validated against the analytical, numerical, and experimental data available in the literature. The validated model is then used to simulate both unsteady- and steady-state flow of an incompressible fluid directly in a representative elementary volume (REV) size micro-CT image of a naturally-occurring sandstone with 3.398 {mu}m resolution. We analyze the quality and consistency of the predicted flow behavior and calculate absolute permeability using the steady-state flow rate.

97 MATHEMATICS AND COMPUTING↗

Extending the length and time scales of Gram–Schmidt Lyapunov vector computations

Lyapunov vectors have found growing interest recently due to their ability to characterize systems out of thermodynamic equilibrium. The computation of orthogonal Gram–Schmidt vectors requires multiplication and QR decomposition of large matrices, which grow as N{sup 2} (with the particle count). This expense has limited such calculations to relatively small systems and short time scales. Here, we detail two implementations of an algorithm for computing Gram–Schmidt vectors. The first is a distributed-memory message-passing method using Scalapack. The second uses the newly-released MAGMA library for GPUs. We compare the performance of both codes for Lennard–Jones fluids from N=100 to 1300 between Intel Nahalem/Infiniband DDR and NVIDIA C2050 architectures. To our best knowledge, these are the largest systems for which the Gram–Schmidt Lyapunov vectors have been computed, and the first time their calculation has been GPU-accelerated. We conclude that Lyapunov vector calculations can be significantly extended in length and time by leveraging the power of GPU-accelerated linear algebra.

97 MATHEMATICS AND COMPUTING↗

Parallel Fokker–Planck-DSMC algorithm for rarefied gas flow simulation in complex domains at all Knudsen numbers

A major challenge for the conventional Direct Simulation Monte Carlo (DSMC) technique lies in the fact that its computational cost becomes prohibitive in the near continuum regime, where the Knudsen number (Kn)—characterizing the degree of rarefaction—becomes small. In contrast, the Fokker–Planck (FP) based particle Monte Carlo scheme allows for computationally efficient simulations of rarefied gas flows in the low and intermediate Kn regime. The Fokker–Planck collision operator—instead of performing binary collisions employed by the DSMC method—integrates continuous stochastic processes for the phase space evolution in time. This allows for time step and grid cell sizes larger than the respective collisional scales required by DSMC. Dynamically switching between the FP and the DSMC collision operators in each computational cell is the basis of the combined FP-DSMC method, which has been proven successful in simulating flows covering the whole Kn range. Until recently, this algorithm had only been applied to two-dimensional test cases. In this contribution, we present the first general purpose implementation of the combined FP-DSMC method. Utilizing both shared- and distributed-memory parallelization, this implementation provides the capability for simulations involving many particles and complex geometries by exploiting state of the art computer cluster technologies.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Multi-rate time integration on overset meshes

Overset meshes are an effective tool for the computational fluid dynamic simulation of problems with complex geometries or multiscale spatio-temporal features. When the maximum allowable timestep on one or more meshes is significantly smaller than on the remaining meshes, standard explicit time integrators impose inefficiencies for time-accurate calculations by requiring that all meshes advance with the smallest timestep. With the targeted use of multi-rate time integrators, separate meshes can be time-marched at independent rates to avoid wasteful computation while maintaining accuracy and stability. This work applies time-explicit multi-rate integrators to the simulation of the compressible Navier-Stokes equations discretized on overset meshes using summation-by-parts (SBP) operators and simultaneous approximation term (SAT) boundary conditions. We introduce a class of multi-rate Adams-Bashforth (MRAB) schemes that offer significant stability improvements and computational efficiencies for SBP-SAT methods. We present numerical results that confirm the efficacy of MRAB integrators, outline a number of implementation challenges, and demonstrate a reduction in computational cost enabled by MRAB. We also look at the use of our method in the setting of a large-scale distributed-memory parallel implementation where we discuss concerns involving load balancing and communication efficiency.

97 MATHEMATICS AND COMPUTING↗

Spectral quadrature for the first principles study of crystal defects: Application to magnesium

In this work, we present an accurate and efficient finite-difference formulation and parallel implementation of Kohn-Sham Density (Operator) Functional Theory (DFT) for non periodic systems embedded in a bulk environment. Specifically, employing non-local pseudopotentials, local reformulation of electrostatics, and truncation of the spatial Kohn-Sham Hamiltonian, and the Linear Scaling Spectral Quadrature method to solve for the pointwise electronic fields in real-space and the non-local component of the atomic force, we develop a parallel finite difference framework suitable for distributed memory computing architectures to simulate non-periodic systems embedded in a bulk environment. Choosing examples from magnesium-aluminum alloys, we first demonstrate the convergence of energies and forces with respect to spectral quadrature polynomial order, and the width of the spatially truncated Hamiltonian. Next, we demonstrate the parallel scaling of our framework, and show that the computation time and memory scale linearly with respect to the number of atoms. Next, we use the developed framework to simulate isolated point defects and their interactions in magnesium-aluminum alloys. Our findings conclude that the binding energies of divacancies, Al solute-vacancy and two Al solute atoms are anisotropic and are dependent on cell size. Furthermore, the binding is favorable in all three cases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Scalable Implicit Solvers with Dynamic Mesh Adaptation for a Relativistic Drift-Kinetic Fokker–Planck–Boltzmann Model

In this work we consider a relativistic drift-kinetic model for runaway electrons along with a Fokker–Planck operator for small-angle Coulomb collisions, a radiation damping operator, and a secondary knock-on (Boltzmann) collision source. Here, we develop a new scalable fully implicit solver utilizing finite volume and conservative finite difference schemes and dynamic mesh adaptivity. A new data management framework in the PETSc library based on the p4est library is developed to enable simulations with dynamic adaptive mesh refinement (AMR), distributed memory parallelization, and dynamic load balancing of computational work. This framework and the runaway electron solver building on the framework are able to dynamically capture both bulk Maxwellian at the low-energy region and a runaway tail at the high-energy region. To effectively capture features via the AMR algorithm, a new AMR indicator prediction strategy is proposed that is performed alongside the implicit time evolution of the solution. This strategy is complemented by the introduction of computationally cheap feature-based AMR indicators that are analyzed theoretically. Numerical results quantify the advantages of the prediction strategy in better capturing features compared with nonpredictive strategies; and we demonstrate trade-offs regarding computational costs. The robustness with respect to model parameters, algorithmic scalability, and parallel scalability are demonstrated through several benchmark problems including manufactured solutions and solutions of different physics models. We focus on demonstrating the advantages of using implicit time stepping and AMR for runaway electron simulations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Parallel streaming between heterogeneous HPC resources for real-time analysis

Performing analysis or generating visualizations concurrently with high performance simulations can yield great benefits compared to post-processing data. Writing and reading large volumes of data can be reduced or eliminated, thereby producing an I/O cost savings. One such method for concurrent simulation and analysis is in transit - streaming data from the resource running the simulation to a separate resource running the analysis. In transit analysis can be beneficial since computational resources may not have certain resources needed for visualization and analysis (e.g. GPUs) and to reduce the impact of performing analysis tasks to the run time of the simulation. When sending and receiving data in transit, data redistribution mechanisms are needed in order to support heterogeneous data layouts that may be required by the simulation and analysis applications. The work described in this paper compares two mechanisms for on-the-fly data redistribution when streaming data in parallel between two distributed memory applications. Finally, our results show that it is often more advantageous to stream data in the same layout as the sender and redistribute data amongst processes on the receiving end than to stream data in the final layout needed by the receiver.

97 MATHEMATICS AND COMPUTING↗

Demonstrating the viability of Lagrangian in situ reduction on supercomputers

Performing exploratory analysis and visualization of large-scale time-varying computational science applications is challenging due to inaccuracies that arise from under-resolved data. In recent years, Lagrangian representations of the vector field computed using in situ processing are being increasingly researched and have emerged as a potential solution to enable exploration. However, prior works have offered limited estimates of the encumbrance on the simulation code as they consider “theoretical” in situ environments. Further, the effectiveness of this approach varies based on the nature of the vector field, benefitting from an in-depth investigation for each application area. With this study, an extended version of Sane et al. (2021), we contribute an evaluation of Lagrangian analysis viability and efficacy for simulation codes executing at scale on a supercomputer. We investigated previously unexplored cosmology and seismology applications as well as conducted a performance benchmarking study by using a hydrodynamics mini-application targeting exascale computing. Here, to inform encumbrance, we integrated in situ infrastructure with simulation codes, and evaluated Lagrangian in situ reduction in representative homogeneous and heterogeneous HPC environments. To inform post hoc accuracy, we conducted a statistical analysis across a range of spatiotemporal configurations as well as a qualitative evaluation. Additionally, our study contributes cost estimates for distributed-memory post hoc reconstruction. In all, we demonstrate viability for each application — data reduction to less than 1% of the total data via Lagrangian representations, while maintaining accurate reconstruction and requiring under 10% of total execution time in over 90% of our experiments.

97 MATHEMATICS AND COMPUTING↗

Streaming data analytics via message passing with application to graph algorithms

The need to process streaming data, which arrives continuously at high-volume in real-time, arises in a variety of contexts including data produced by experiments, collections of environmental or network sensors, and running simulations. Streaming data can also be formulated as queries or transactions which operate on a large dynamic data store, e.g. a distributed database. We describe a lightweight, portable framework named PHISH which enables a set of independent processes to compute on a stream of data in a distributed-memory parallel manner. Datums are routed between processes in patterns defined by the application. PHISH can run on top of either message-passing via MPI or sockets via ZMQ. The former means streaming computations can be run on any parallel machine which supports MPI; the latter allows them to run on a heterogeneous, geographically dispersed network of machines. We illustrate how PHISH can support streaming MapReduce operations, and describe streaming versions of three algorithms for large, sparse graph analytics: triangle enumeration, subgraph isomorphism matching, and connected component finding. Lastly, we also provide benchmark timings for MPI versus socket performance of several kernel operations useful in streaming algorithms.

97 MATHEMATICS AND COMPUTING↗

A communication-avoiding 3D algorithm for sparse LU factorization on heterogeneous systems

We propose a new algorithm to improve the strong scalability of right-looking sparse LU factorization on distributed memory systems. Our 3D algorithm for sparse LU uses a three-dimensional MPI process grid, exploits elimination tree parallelism, and trades off increased memory for reduced per-process communication. We also analyze the asymptotic improvements for planar graphs (e.g., those arising from 2D grid or mesh discretizations) and certain non-planar graphs (specifically for 3D grids and meshes). For a planar graph with $n$ vertices, our algorithm reduces communication volume asymptotically in $n$ by a factor of $\mathscr{O}$ $\Big(\sqrt{log \ n}\Big)$ and latency by a factor of $\mathscr{O}$ $(log \ n)$. For nonplanar cases, our algorithm can reduce the per-process communication volume by 3× and latency by $\mathscr{O}$ $\Big(n^\frac13\Big)$ times. In all cases, the memory needed to achieve these gains is a constant factor. We implemented our algorithm by extending the 2D data structure used in SuperLU_DIST. Our new 3D code achieves empirical speedups up to 27× for planar graphs and up to 3.3× for non-planar graphs over the baseline 2D SuperLU_DIST when run on 24,000 cores of a Cray XC30. We extend the 3D algorithm for heterogeneous architectures by adding the Highly Asynchronous Lazy Offload (Halo) algorithm for co-processor offload. On 4096 nodes of a Cray XK7 with 32,768 CPU cores and 4096 Nvidia K20x GPUs, the 3D algorithm achieves empirical speedups up to 24× for planar graphs and 3.5× for non-planar graphs over the baseline 2D SuperLU_DIST with co-processor acceleration.

97 MATHEMATICS AND COMPUTING↗

Online multimedia retrieval on CPU–GPU platforms with adaptive work partition

Nearest neighbors search is a core operation found in several online multimedia services. These services have to handle very large databases, while, at the same time, they must minimize the query response times observed by users. This is specially complex because those services deal with fluctuating query workloads (rates). Consequently, they must adapt at run-time to minimize the response times as the load varies. In this paper, we address the aforementioned challenges with a distributed memory parallelization of the product quantization nearest neighbor search, also known as IVFADC, for hybrid CPU–GPU machines. Overall, our parallel IVFADC implements an out-of-GPU memory execution scheme to use the GPU for databases in which the index does not fit in its memory, which is crucial for searching in very large databases. The careful use of CPU and GPU with work stealing led to an average response time reduction of 2.4 as compared to using the GPU only. Also, our approach to adapt the system to fluctuating loads, called Dynamic Query Processing Policy (DQPP), attained a response time reduction of up to 5 vs. the best static (BS) policy for moderate loads. The system has attained high query processing rates and near-linear scalability in all experiments. We have evaluated our system on a machine with up to 256 NVIDIA V100 GPUs processing a database of 256 billion SIFT features vectors.

97 MATHEMATICS AND COMPUTING↗

Massively-parallel Lagrangian particle code and applications

Massively-parallel, distributed-memory algorithms for the Lagrangian particle hydrodynamic method (Samulyak et al., 2018) have been developed, verified, and implemented. The key component of parallel algorithms is a particle management module that includes a parallel construction of octree databases, dynamic adaptation and refinement of octrees, and particle migration between parallel subdomains. The particle management module is based on the p4est (parallel forest of k-trees) library. The massively-parallel Lagrangian particle code has been applied to a variety of fundamental science and applied problems. A summary of Lagrangian particle code applications to the injection of impurities into thermonuclear fusion devices and to the simulation of supersonic hydrogen jets in support of laser-plasma wakefield acceleration research has also been presented.

97 MATHEMATICS AND COMPUTING↗

Scalable self attraction and loading calculations for unstructured ocean tide models

Self attraction and earth-loading effects are important for accurately modeling global tides. A common approach of handling this forcing is to expand mass anomalies into spherical harmonics, which are scaled by load Love numbers to account for elastic earth deformation. We investigate two different approaches to perform these calculations for ocean models that employ unstructured meshes and distributed memory parallelization. The first approach leverages a highly efficient spherical harmonics library, but requires all-to-one and one-to-all communications and interpolation operations between the unstructured and a structured mesh. This approach is compared to a parallel algorithm that computes the spherical harmonic transformations directly on the unstructured mesh with an all-reduce communication. Here, our results show that although the unstructured mesh calculations are more expensive, the scalability of the unstructured mesh approach allows for more efficient spherical harmonics transforms for high-resolution meshes and large processor counts. This methodology enables the efficient inclusion of tidal dynamics large-scale Earth system model simulations.

54 ENVIRONMENTAL SCIENCES↗