Maximizing available hardware parallelism on CPU and GPU architectures for finite element applications
Explore the source record for details and available documents.
Mosby, Matthew David [Sandia National Laboratories↗
SEARCH · Search NASA
Results for “GPU finite elements”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A Performance Portable, Fully Implicit Landau Collision Operator with Batched Linear Solvers
Modern accelerators use hierarchical parallel programming models that enable massive multithreading within a processing element (PE), with multiple PEs per device driven by traditional processes. Batching is a technique for exposing PE-level parallelism in algorithms that have traditionally run on MPI processes or multiple threads within a single process. Opportunities for batching arise in, for example, kinetic discretizations of magnetized plasmas where collisions are advanced in velocity space at each spatial point independently. This paper builds on previous work on a high-performance, fully nonlinear, Landau collision operator by batching the linear solver, as well as batching the spatial point problems and adding new support for multiple grids for multiscale, multispecies problems. An anisotropic relaxation verification test that agrees well with previously published results and analytical models is presented. The performance results from NVIDIA A100 and AMD MI250X nodes are presented with hardware utilization analysis for each architecture. Finally, the entire implicit Landau operator time advance is implemented in Kokkos for performance portability, running entirely on the device and is available in the PETSc numerical library.
97 MATHEMATICS AND COMPUTING↗
A GPU-Enabled Higher Order Mixed Finite Element Method for Pulsed Power ICF Target Design
Explore the source record for details and available documents.
Physics - Plasma physics↗
Real-Time Bayesian Inference at Extreme Scale: A Digital Twin for Tsunami Early Warning Applied to the Cascadia Subduction Zone
We present a Bayesian inversion-based digital twin that employs acoustic pressure data from seafloor sensors, along with 3D coupled acoustic–gravity wave equations, to infer earthquake-induced spatiotemporal seafloor motion in real time and forecast tsunami propagation toward coastlines for early warning with quantified uncertainties. Our target is the Cascadia subduction zone, with one billion parameters. Computing the posterior mean alone would require 50 years on a 512 GPU machine. Instead, exploiting the shift invariance of the parameter-to-observable map and devising novel parallel algorithms, we induce a fast offline–online decomposition. The offline component requires just one adjoint wave propagation per sensor; using MFEM, we scale this part of the computation to the full El Capitan system (43,520 GPUs) with 92% weak parallel efficiency. Moreover, given real-time data, the online component exactly solves the Bayesian inverse and forecasting problems in 0.2 seconds on a modest GPU system, a ten-billion-fold speedup.
97 MATHEMATICS AND COMPUTING↗
Benchmarking of massively parallel phase-field codes for directional solidification
We present a detailed benchmark comparing two state-of-the-art phase-field implementations for simulating alloy solidification under experimentally relevant conditions. The study investigates the directional solidification of Al-3wt%Cu under high-velocity solidification conditions and SCN-0.46wt% camphor under microgravity conditions from National Aeronautics and Space Administration (NASA) DECLIC-DSI-R experiments. Both codes, one employing finite-difference discretization with uniform mesh and GPU-acceleration (GPU-PF) and the other one employing finite-element discretization with adaptive-mesh and CPU-parallelization (PRISMS-PF), solve the same quantitative phase-field formulation that incorporates an anti-trapping current for the solidification of dilute alloys. We evaluate the predictions of each code for dendritic morphology, primary spacing, and tip dynamics in both 2D and 3D, as well as their numerical convergence and computational performance. While existing benchmark problems have primarily focused on simplified or small-scale simulations, they do not reflect the computational and modeling challenges posed by employing experimentally relevant time and length scales. Our results provide a practical framework for assessing phase-field code performance as well as validating and facilitating their application in integrated computational materials engineering (ICME) workflows that require integration with realistic experimental data.
36 MATERIALS SCIENCE↗
Optimization of a Solver for Computational Materials and Structures Problems on NVIDIA Volta and AMD Instinct GPUs
The Scalable Implementation of Finite Elements by NASA (ScIFEN) is a software package developed to solve complex computational materials and structures problems using the finite element method (FEM). In this paper, we describe optimization techniques to speed up the linear solver computation that occurs within the ScIFEN application. We consider GPUs from two different vendors, NVIDIA and AMD as our target platforms for optimization and highlight differences in performance and optimization techniques. The NVIDIA GPU Volta V100 is used in the Summit system deployed at Oak Ridge National Laboratory, and the new exascale system, Frontier, will be using AMD Radeon Instinct GPU. We evaluated the performance of various optimization techniques on test matrices, ranging in size from100K to 4M, that are representative of ScIFEN applications. The linear solver computation is memory-bound on both GPUs. Our experiments show that on the NVIDIA GPU we obtained up to79%of the theoretical peak bandwidth, while the AMD GPU achieved 59%. Overall, the NVIDIA V100 GPU outperforms the AMD MI 25 GPU1. We observed an overall speedup of up to37X on an NVIDIA V100 compared to an Intel Skylake 12-coremachine. The solver for a 4M degree of freedom system took under 2.5 seconds.
Mohammad Zubair↗
Assessment of Edge-Based Viscous Method for Corner-Flow Solutions on Graphics Processing Units
A highly efficient, edge-based viscous (EBV) discretization method has been recently implemented in a practical, unstructured-grid, node-centered, finite-volume flow solver and evaluated for Reynolds-averaged Navier-Stokes (RANS) formulations. In comparison to a well-established cell-based viscous (CBV) method, the EBV method has demonstrated multifold acceleration of all viscous-kernel computations on general unstructured mixed-element grids. The viscous kernels include evaluation of viscous fluxes, diffusion terms in turbulence models, and the corresponding Jacobian terms. In this paper, an EBV implementation of a nonlinear extension of the Spalart-Allmaras turbulence model, SA-neg-QCR2000, is presented and verified. The SA-neg-QCR2000 model is used for simulating turbulent corner flows. Previously reported EBV computations have been conducted on traditional computing architectures based on central processing units (CPU). This paper assesses benefits of the EBV method on modern high-performance computing architectures based on graphics processing units (GPU). The GPU implementations of the CBV and EBV methods are verified by comparing solutions and iterative convergence with those observed in CPU computations on the same grids. A comprehensive assessment of the EBV speedup on CPU and GPU architectures is presented for established benchmark corner flows, namely, a supersonic flow through a long square duct and a subsonic flow around a NASA juncture flow model.
CFD↗
Assessment of Edge-Based Viscous Method for Corner-Flow Solutions on Graphics Processing Units
A highly efficient, edge-based viscous (EBV) discretization method has been recently implemented in a practical, unstructured-grid, node-centered, finite-volume flow solver and evaluated for Reynolds-averaged Navier-Stokes (RANS) formulations. In comparison to a well-established cell-based viscous (CBV) method, the EBV method has demonstrated multifold acceleration of all viscous-kernel computations on general unstructured mixed-element grids. The viscous kernels include evaluation of viscous fluxes, diffusion terms in turbulence models, and the corresponding Jacobian terms. In this paper, an EBV implementation of a nonlinear extension of the Spalart-Allmaras turbulence model, SA-neg-QCR2000, is presented and verified. The SA-neg-QCR2000 model is used for simulating turbulent corner flows. Previously reported EBV computations have been conducted on traditional computing architectures based on central processing units (CPU). This paper assesses benefits of the EBV method on modern high-performance computing architectures based on graphics processing units (GPU). The GPU implementations of the CBV and EBV methods are verified by comparing solutions and iterative convergence with those observed in CPU computations on the same grids. A comprehensive assessment of the EBV speedup on CPU and GPU architectures is presented for established benchmark corner flows, namely, a supersonic flow through a long square duct and a subsonic flow around a NASA juncture flow model.
CFD↗
Direct sensitivity analysis on the parameterization of crystal plasticity models
Various methods for calibrating crystal plasticity finite element (CPFE) models lead to non-unique input parameter values, which subsequently introduce uncertainty in the predicted mechanical response. Sensitivity analysis (SA) conducted on crystal plasticity models is used to identify how variability in these parameters contribute to output uncertainty. Traditional SA on CPFE parameters uses simplified surrogate models to save computational time. However, the accuracy of the surrogate models depends on the quantity of training data used, and any modeling error can propagate into the SA results, potentially affecting their reliability. In this work, the elementary effects test (EET) method, a global SA technique using direct CPFE simulations was employed, and the results obtained were compared with the First Order Second Moment (FOSM) method. ExaConstit, an open-source GPU-enabled CPFE code, was used to perform the simulations and direct SA. The EET method was accurately able to capture the non-linear effects of all the input parameters on the output and is a valuable approach for reliably attributing parameter sensitivities in CPFE models. Based on the results, efficient strategies to perform future parameter calibration and SA are discussed. Additionally, the SA trends observed in different single crystal orientations closely mirrored the activity of the slip systems.
Elementary Effects Test↗
Memory-Aware External Facelist Calculation: A Data-Parallel Atomic Hash Counting Approach
Unstructured volumetric meshes serve as fundamental data representations in various scientific simulations and analyses. They play a crucial role in representing complex computational domains and are essential for important numerical techniques, such as finite element analysis. Whenever such a mesh is read from a file, streamed in-situ, or generated by algorithms, scientific visualization libraries rely on calculating the external surface of a geometry, named “external facelist”, to produce a polygonal mesh for rendering. Consequently, external facelist calculation has become one of the most widely used algorithms in the scientific visualization domain, necessitating optimal performance. In this paper, we explore relevant work on external facelist calculation algorithms in two common visualization libraries, VTK and Viskores, assess their performance and memory constraints, and introduce a novel memory-aware external facelist calculation algorithm employing an atomic hash counting approach. This algorithm fully leverages Viskores' data-parallel primitive operations, facilitating its execution across diverse many-core architectures. Our algorithm features the lowest memory footprint on the GPU and the second-lowest on the CPU among all evaluated methods, and it also delivers the fastest performance on both CPU and GPU. It has been made available under an open-source license in the VTK and Viskores visualization systems.
Tsalikis, Spiros [Kitware] (ORCID:0000000151137195↗
Optimal Polynomial Smoothers and One‐Sided V‐Cycles for Poisson Problems
The solution to the Poisson equation arising from the spectral element discretization of the incompressible Navier‐Stokes equations needs robust preconditioning strategies. One such strategy is multigrid. To realize the potential of multigrid methods, effective smoothing strategies are needed. Chebyshev polynomial smoothers, in conjunction with pointwise Jacobi or additive Schwarz methods (ASMs), prove to be an effective smoother. Other polynomial smoothers, however, may provide superior convergence to the multigrid preconditioner. The authors compare the standard Chebyshev polynomial smoothers to both the novel fourth‐kind Chebyshev polynomial smoothers proposed by Lottes as well as smoothers based on the polynomial of best uniform approximation to as proposed by Kraus, Vassilevski, and Zikatanov. At the cost of symmetry, further improvements may be made. For example, a order polynomial smoother on both sides of the V‐cycle may be substituted with an order polynomial smoother on one side at no additional cost. The choice of omitting the postsmoother in favor of higher‐order polynomial presmoothing is advantageous in cases where the multigrid approximation property constant is large. The authors consider a 2D model problem based on finite differences to motivate the choice of polynomial smoother, order, and whether to apply postsmoothing for the target application of high‐order ‐geometric multigrid methods for GPU architectures. Results from both domains demonstrate the substantial improvement of these approaches over the standard Chebyshev polynomial smoother with a symmetric V‐cycle.
97 MATHEMATICS AND COMPUTING↗
Multi-material ALE remap with interface sharpening using high-order matrix-free finite element methods
The arbitrary Lagrangian-Eulerian (ALE) technique involves remapping field quantities from a Lagrangian mesh to an optimized mesh in a conservative, accurate and bounds-preserving manner. For methods based on arbitrary order finite elements, as described in a reference, material volume fractions are advected in pseudo-time using flux-corrected transport (FCT) without any form of interface reconstruction. In practice, this can lead to excessive propagation of small volume fractions throughout the domain. In addition, this method requires assembly of a global advection matrix to compute the bounds-preserving low-order FCT solution. In this work, we introduce a new approach for ALE remap using a high-order matrix-free technique which incorporates a flux modification to sharpen material interfaces in a conservative manner. Our approach begins with computing a bounds-preserving low-order solution to the ALE remap equations at the element level. We then compute a sharp interface solution (not guaranteed to be bounds-preserving) which comes from solving an augmented version of the ALE remap equations with a conservative flux modification which acts to sharpen material volume fractions based on their gradients and transport directions. Using the sharp interface solution, we make global corrections to the bounds-preserving solution while maintaining preservation of bounds. By blending with the sharpened solution at the global level we are able to globally conserve mass without hindering the remap pseudo-time step. This new interface-aware ALE remap method is based entirely on partial assembly techniques where globally assembled matrix operators are no longer needed, resulting in a globally matrix-free FCT method for multi-material, multi-field ALE remap with high performance on GPU architectures. We present results of our new remap method on 1D, 2D and 3D benchmarks and describe the algorithmic tailoring for GPU architectures that was developed.
Vargas, Arturo [Lawrence Livermore National Labora↗