Preparing an Incompressible-Flow Fluid Dynamics Code for Exascale-Class Wind Energy Simulations: Preprint

Mullowney, Paul (ORCID:0000000215045178); Li, Ruipeng (ORCID:0000000328025763); Thomas, Stephen; Ananthan, Shreyas; Sharma, Ashesh (ORCID:0000000333379160); Rood, Jon S. (ORCID:0000000275133225); Williams, Alan B.; Sprague, Michael A. (ORCID:0000000305190379)

DOE OSTI · 1823595

Preparing an Incompressible-Flow Fluid Dynamics Code for Exascale-Class Wind Energy Simulations: Preprint

Mullowney, Paul (ORCID:0000000215045178)·Li, Ruipeng (ORCID:0000000328025763)·Thomas, Stephen·Ananthan, Shreyas·Sharma, Ashesh (ORCID:0000000333379160)·Rood, Jon S. (ORCID:0000000275133225)·Williams, Alan B.·Sprague, Michael A. (ORCID:0000000305190379)

View original source ↗Read source document ↗Export metadata ↓

Abstract

The US Department of Energy has identified Exascale-Class wind farm simulation tools as critical to wind energy scientific discovery. A primary objective of the Exawind project is to build high-performance, predictive Computational Fluid Dynamics tools that satisfy these modeling needs. GPU accelerators will serve as the computational thoroughbreds of next generation, Exascale-Class, platforms. Here, we report on our efforts for preparing the Exawind unstructured mesh solver, Nalu-Wind, for Exascale-Class machines. For computing at this scale, a simple port of the incompressible-flow algorithms to GPUs is not sufficient. One needs novel algorithms that are application aware, memory efficient, and optimized for latest generation GPU devices to get high-performance. The result of our efforts are unstructured mesh simulations of wind turbines that use 1/6 the compute resources of Summit supercomputer at Oak Ridge National Lab. In particular, we demonstrate a first-of-its-kind, simulation using Algebraic Multigrid solvers on over 4000 GPUs.

Explore related subjects

algebraic multigrid CFD computational linear algebra exascale GPU computing MATHEMATICS AND COMPUTING,WIND ENERGY

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Mullowney, Paul (ORCID:0000000215045178), Li, Ruipeng (ORCID:0000000328025763), Thomas, Stephen, Ananthan, Shreyas, Sharma, Ashesh (ORCID:0000000333379160), Rood, Jon S. (ORCID:0000000275133225), Williams, Alan B., Sprague, Michael A. (ORCID:0000000305190379). 2021-09-28. Preparing an Incompressible-Flow Fluid Dynamics Code for Exascale-Class Wind Energy Simulations: Preprint. https://www.osti.gov/biblio/1823595

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

DOE OSTI2023

Scaled ILU Smoothers for Navier-Stokes Pressure Projection

Incomplete LU (ILU) smoothers are effective in the algebraic multigrid (AMG) V-cycle for reducing high-frequency components of the error. However, the requisite direct triangular solves are comparatively slow on GPUs. Previous work has demonstrated the advantages of Jacobi iteration as an alternative to direct solution of these systems. Depending on the threshold and fill-level parameters chosen, the factors can be highly nonnormal and Jacobi is unlikely to converge in a low number of iterations. We demonstrate that row scaling can reduce the departure from normality, allowing us to replace the inherently sequential solve with a rapidly converging Richardson iteration. There are several advantages beyond the lower compute time. Scaling is performed locally for a diagonal block of the global matrix because it is applied directly to the factor. Further, an ILUT Schur complement smoother maintains a constant GMRES iteration count as the number of MPI ranks increases, and thus parallel strong-scaling is improved. Our algorithms have been incorporated into hypre, and we demonstrate improved time to solution for linear systems arising in the Nalu-Wind and PeleLM pressure solvers. For large problem sizes, GMRES+AMG executes at least five times faster when using iterative triangular solves compared with direct solves on massively parallel GPUs.

algebraic multigrid↗

DOE OSTI2022

Two-Stage Gauss-Seidel Preconditioners and Smoothers for Krylov Solvers on a GPU Cluster: Preprint

Gauss-Seidel (GS) relaxation is often employed as a preconditioner for a Krylov solver or as a smoother for Algebraic Multigrid (AMG). However, the requisite sparse triangular solve is difficult to parallelize on many-core architectures such as graphics processing units (GPUs). In the present study, the performance of the sequential GS relaxation based on a triangular solve is compared with two-stage variants, replacing the direct triangular solve with a fixed number of inner Jacobi-Richardson (JR) iterations. When a small number of inner iterations is sufficient to maintain the Krylov convergence rate, the two-stage GS (GS2) often outperforms the sequential algorithm on many-core architectures. The GS2 algorithm is also compared with JR. When they perform the same number of ops for SpMV (e.g. three JR sweeps compared to two GS sweeps with one inner JR sweep), the GS2 iterations, and the Krylov solver preconditioned with GS2, may converge faster than the JR iterations. Moreover, for some problems (e.g. elasticity), it was found that JR may diverge with a damping factor of one, whereas two-stage GS may improve the convergence with more inner iterations. Finally, to study the performance of the two-stage smoother and preconditioner for a practical problem, these were applied to incompressible uid ow simulations on GPUs.

algebraic multigrid↗

DOE OSTI2021

Preparing an Incompressible-Flow Fluid Dynamics Code for Exascale-Class Wind Energy Simulations

The U.S. Department of Energy has identified exascale-class wind farm simulation as critical to wind energy scientific discovery. A primary objective of the ExaWind project is to build high-performance, predictive computational fluid dynamics (CFD) tools that satisfy these modeling needs. GPU accelerators will serve as the computational thoroughbreds of next-generation, exascale-class supercomputers. Here, we report on our efforts in preparing the ExaWind unstructured mesh solver, Nalu-Wind, for exascale-class machines. For computing at this scale, a simple port of the incompressible-flow algorithms to GPUs is insufficient. To achieve high performance, one needs novel algorithms that are application aware, memory efficient, and optimized for the latest-generation GPU devices. The result of our efforts are unstructured-mesh simulations of wind turbines that can effectively leverage thousands of GPUs. In particular, we demonstrate a first-of-its-kind, incompressible-flow simulation using Algebraic Multigrid solvers that strong scales to more than 4000 GPUs on the Summit supercomputer.

algebraic multigrid↗