High Fidelity Large Eddy Simulation of Reactive Supercritial Fuel Jet-in-Cross-Flow using GPU Acceleration
Explore the source record for details and available documents.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Abstract not provided.
Abstract not provided.
Abstract not provided.
Abstract not provided.
Explore the source record for details and available documents.
Accurate solutions of the radiation transport equation play an important role in the analysis of many nuclear systems. Because there is no need to discretize the problem, Monte Carlo methods offer the most accurate possible solutions to the transport equation. However, significant computational effort is required to reduce the stochastic noise inherent to Monte Carlo methods. Current trends in high performance computing are moving towards vectorized, single instruction multiple data (SIMD), architectures such as Intel Xeon Phis and graphics processing units (GPUs) due to their favorable ratio of performance to power consumption. The challenge of adapting Monte Carlo transport to vectorized computing architectures was first addressed by Brown with an algorithm known as event-based Monte Carlo. In this approach, rather than simulating individual particle histories from birth to death, groups of particles are processed one event at a time. This allows each event batch to perform the same operations in lockstep, enabling use of the vector hardware. Although modern vectorized hardware is more versatile than the early vector machines, reducing so-called thread divergence is nonetheless still an important consideration for algorithm design. Many of the algorithms proposed so far for Monte Carlo transport on advanced architectures have focused on variants of event-based transport. Some evidence, however, suggests that low-level thread divergence, resulting from short-lived branching statements, may not be detrimental to overall performance on GPUs as long as the effect on achievable memory bandwidth is considered. This paper characterizes the performance of a traditional history-based Monte Carlo algorithm on GPUs. We show that with only minor modifications to traditional algorithms, significant performance gains relative to CPUs are possible. (authors)
Accelerated computing in HPC such as with GPU, plays a central role in HPC nowadays. However, in some complicated applications with partially different performance behavior is hard to solve with a single type of accelerator where GPU is not the perfect solution in these cases. We are developing a framework and transpiler allowing the users to program the codes with a single notation of OpenACC to be compiled for multi-hybrid accelerators, named MHOAT (Multi-Hybrid OpenACC Translator) for HPC applications. MHOAT parses the original code with directives to identify the target accelerating devices, currently supporting NVIDIA GPU and Intel FPGA, dispatching these specific partial codes to background compilers such as NVIDIA HPC SDK for GPU and OpenARC research compiler for FPGA, then assembles binaries for the final object with FPGA bitstream file. In this paper, we present the concept, design, implementation, and performance evaluation of a practical astrophysics simulation code where we successfully enhanced the performance up to 10 times faster than the GPU-only solution.
The simulation of nuclear transients using Computational Fluid Dynamics (CFD) presents significant computational challenges due to the inherent complexity and the wide separation in temporal scales between various flow physical phenomena. These disparities lead to high computational costs, often making the simulation of transients impractical without advanced techniques. Consequently, multiple research initiatives are being pursued by the NEAMS thermal-hydraulic area, some driven by academic institutions and some by national laboratories. Overall, they are exploring novel methods to make transient simulations more feasible and efficient. This report delves into recent advancements within the CFD code NekRS, specifically those achieved in Fiscal Year 2024 under the CONNECT effort, aimed at improving the performance and feasibility of transient simulations. The first major advancement involves the porting of NekRS to Aurora, one of the Department of Energy’s (DOE) most powerful supercomputers. Additionally, the report discusses the implementation of an overlapping domain capability within NekRS. This novel GPU-accelerated capability allows different spatial regions of the domain to be solved independently, enhancing the code’s efficiency, particularly when running large-scale simulations in complex domains. The scalability of this approach is demonstrated, highlighting its potential to transform how transients are approached in CFD simulations. Lastly, the report focuses on how this overlapping domain capability specifically accelerates transient simulations through multi-rate timestepping. By decoupling different regions and facilitating faster computations, this method offers a promising pathway to making nuclear transient simulations more computationally feasible, addressing one of the critical bottlenecks in the field. Together, these advancements represent a significant leap forward in transient simulation technology, bringing closer the possibility of handling highly complex nuclear scenarios with greater efficiency and accuracy.
Computational bioinformatics and biomedical applications frequently contain heterogeneously sized units of work or tasks, for instance due to variability in the sizes of biological sequences and molecules. Variable-sized workloads lead to load imbalances in parallel implementations which detract from efficiency and performance. Many modern computing resources now have multiple graphics processing units(GPUs) per computer for acceleration. These multiple GPU resources need to be used efficiently through balancing of workloads across the GPUs. OpenMP is a portable directive-based parallel programming API used ubiquitously in bioscience applications to program CPUs; recently, the use of OpenMP directives for GPU acceleration has become possible. Here, motivated by experiences with imbalanced loads in GPU-accelerated bioinformatics applications, we address the load balancing problem using OpenMP task-to-GPU scheduling combined with OpenMP GPU offloading for multiply heterogeneous workloads – loads with both variable input sizes, and simultaneously, variable convergence rates for algorithms with a stochastic component – scheduled across multiple GPUs. We aim to develop strategies which are both easy to use and have lower overheads, and may be incorporated incrementally in existing programs which already make use of OpenMP for CPU-based threading in order to make use of multi-GPU computers. We test different combinations of input size variability and convergence rate variability, and characterize the effects of these different scenarios on the performance of scheduling strategies across multiple GPUs with OpenMP. We present several dynamic scheduling solutions for different parallel patterns, explore optimizations, and provide publicly available example computational kernels to make these strategies easy to use in programs. This work will enable application developers to efficiently and easily use multiple GPUs for imbalanced workloads found in bioinformatics and biomedical applications.
Improving the understanding of how computational fluid dynamics (CFD) direct numerical simulations (DNS) of flows in the High Flux Isotope Reactor (HFIR) perform when run in parallel using the high performance computing (HPC) platform Summit at the Oak Ridge Leadership Computing Facility (OLCF) is of particular importance to boost the computational tools used to support HFIR conversion to low enriched fuel (LEU). Evaluation of scaling performance was driven by the increasing importance of graphics processing unit (GPU) usage in HPC, which is becoming the standard for modern supercomputers such as Summit. The desired results are to obtain a strong positive correlation between the computational resources dedicated to a problem and the relative speed-up of the simulation in comparison to a benchmark. This capability will allow substantially improvement in HFIR flow analytical capabilities, specifically when predicting turbulence properties at high Reynolds numbers. The study leverages previous simulation results performed with code PHASTA (finite element) on HPC platforms Cori (NERSC) and Theta (ALCF) [1] with computing options provided in the computing platform OpenFOAM (finite volume) at OLCF. Transitioning from PHASTA to OpenFOAM will (1) eliminate dependence on third-party software for mesh generation and manipulation, (2) reduce resource needs by employing modern architectures, and (3) build expertise for future modeling of HFIR-specific problems like heat transfer in involute geometry, entrance effects, flow structure in channel corners, and so on—all important issues when defining the available thermal margins in the transition to LEU. CPUs and GPUs differ significantly in their architecture and utilization, as discussed in the literature [2]. The most important differences are in the approach to computations and their memory. A single GPU contains a large quantity of cores, enabling it to perform with a much higher throughput than a CPU, but execution requires a different approach. GPU codes execute instructions using the Single-Instruction Multiple-Thread (SIMT) approach in which a single instruction is used for groups of threads called warps. A warp typically consists of 32 threads which must execute the same set of instructions, although on separate threads. Alternately, a CPU has far fewer cores that are much more flexible in their operation, excelling at quickly performing more complex serial computations. This is why GPUs have greater throughput when properly utilized. The second important difference is seen when comparing their memory spaces. Limited memory allocations and CPU–GPU communications cause a significant bottleneck in GPU-accelerated programs. Further study was required to properly take advantage of GPU resources. A comprehensive analysis of code performance and the model-specific features of turbulence constitutes the core of this work. In this study, a DNS simulation of HFIR channel turbulence was performed with the finite volume CFD code OpenFOAM v2112 and CUDA v11.0 on Red Hat Enterprise Linux v8.2. The OpenFOAM installation had AMGx integrated to enable GPU acceleration and utilizes the PETSc4FOAM library. The computational resources and the problem size were scaled on CPU and CPU + GPU architectures to gain a better understanding of the performance of a DNS problem on modern computing hardware. The study aimed to analyze the scaling of the code exclusively on CPUs and then to examine the scaling of the codes with GPU acceleration enabled. Scaling studies included CPU and GPU acceleration on a mesh of varying resolution to analyze the impact of problem size relative to computational resources. In the course of preparing the GPU configuration on Summit, mainly using the AMGX solvers, difficulties were encountered stemming from constant changes resulting from extensive ongoing development activities and the changing environment. This resulted in the inability to complete the GPU portion of the work. The code was compiled and tested, but production runs to assess acceleration were not performed because the used discretional compute time allocation expired as year-end approached. The Summit HPC platform is scheduled for decommissioning in 2024, making it unattractive for future use with Nvidia-based GPUs. Therefore, the work will be moved onto NERSC machines in FY24. An application was prepared and submitted, and sufficient node-hours were awarded to continue the research in the next calendar year. This report summarizes work performed thus far, which mostly focused on CPU OpenFOAM computing.
Rapidly changing computer architectures, such as those found at high-performance computing (HPC) facilities, present the need for mini-applications (miniapps) that capture essential algorithms used in large applications to test program performance and portability, aiding transitions to new systems. The COVID-19 pandemic has fueled a flurry of activity in computational drug discovery, including the use of supercomputers and GPU acceleration for massive virtual screens for therapeutics. Recent work targeting COVID-19 at the Oak Ridge Leadership Computing Facility (OLCF) used the GPU-accelerated program AutoDock-GPU to screen billions of compounds on the Summit supercomputer. In this paper we present the development of a new miniapp, miniAutoDock-GPU, that can be used to evaluate the performance and portability of GPU-accelerated protein-ligand docking programs on different computer architectures. These tests are especially relevant as facilities transition from petascale systems and prepare for upcoming exascale systems that will use a variety of GPU vendors. The key calculations, namely, the Lamarckian genetic algorithm combined with a local search using a Solis-Wets based random optimization algorithm, are implemented. We developed versions of the miniapp using several different programming models for GPU acceleration, including a version using the CUDA runtime API for NVIDIA GPUs, and the Kokkos middle-ware API which is facilitated by C++ template libraries. A third version, currently in progress, uses the HIP programming model. These efforts will help facilitate the transition to exascale systems for this important emerging HPC application, as well as its use on a wide range of heterogeneous platforms.
We present a detailed benchmark comparing two state-of-the-art phase-field implementations for simulating alloy solidification under experimentally relevant conditions. The study investigates the directional solidification of Al-3wt%Cu under high-velocity solidification conditions and SCN-0.46wt% camphor under microgravity conditions from National Aeronautics and Space Administration (NASA) DECLIC-DSI-R experiments. Both codes, one employing finite-difference discretization with uniform mesh and GPU-acceleration (GPU-PF) and the other one employing finite-element discretization with adaptive-mesh and CPU-parallelization (PRISMS-PF), solve the same quantitative phase-field formulation that incorporates an anti-trapping current for the solidification of dilute alloys. We evaluate the predictions of each code for dendritic morphology, primary spacing, and tip dynamics in both 2D and 3D, as well as their numerical convergence and computational performance. While existing benchmark problems have primarily focused on simplified or small-scale simulations, they do not reflect the computational and modeling challenges posed by employing experimentally relevant time and length scales. Our results provide a practical framework for assessing phase-field code performance as well as validating and facilitating their application in integrated computational materials engineering (ICME) workflows that require integration with realistic experimental data.
Abstract Feature selection is an important step in machine learning that can benefit from GPU acceleration. As the number of GPU vendors increases, it is imperative to adapt algorithms such as the minimum Redundancy Maximum Relevance (mRMR) feature selection method to different backends that support several GPU architectures. This work presents a multi-backend implementation of mRMR across CUDA, HIP, and SYCL, and studies its performance when combined with Bayesian optimization and transfer learning to automatically tune execution parameters for different platforms and datasets. Our experimental results show that when tuned, CUDA and HIP achieve comparable performance on NVIDIA architectures, while SYCL exhibits a moderate performance gap. Overall, this work highlights the impact of backend choice and autotuning on GPU-accelerated feature selection and provides insights into deploying mRMR across heterogeneous environments.
We present hipBone, an open-source performance-portable proxy application for the Nek5000 (and NekRS) computational fluid dynamics applications. HipBone is a fully GPU-accelerated C++ implementation of the original NekBone CPU proxy application with several novel algorithmic and implementation improvements which optimize its performance on modern fine-grain parallel GPU accelerators. Our optimizations include a conversion to store the degrees of freedom of the problem in assembled form in order to reduce the amount of data moved during the main iteration and a portable implementation of the main Poisson operator kernel. We demonstrate near-roofline performance of the operator kernel on three different modern GPU accelerators from two different vendors. We present a novel algorithm for splitting the application of the Poisson operator on GPUs which aggressively hides MPI communication required for both halo exchange and assembly. Our implementation of nearest-neighbor MPI communication then leverages several different routing algorithms and GPU-Direct RDMA capabilities, when available, which improves scalability of the benchmark. We demonstrate the performance of hipBone on three different clusters housed at Oak Ridge National Laboratory, namely, the Summit supercomputer and the Frontier early-access clusters, Spock and Crusher. Our tests demonstrate both portability across different clusters and very good scaling efficiency, especially on large problems.
This paper presents a comprehensive performance and scaling analysis of direct numerical simulations for reacting boundary layers, focusing on slab burner configurations. Using a PETSc-based finite volume CFD framework, the study evaluates the scalability and computational cost of flow, chemistry, and radiation evaluations across 2D and 3D simulations. Polymethyl methacrylate (PMMA) is the fuel with pure O 2 as the oxidizer, modeled using a detailed chemical kinetics mechanism with 113 species and 660 reactions. A ray-tracing-based radiation solver, designed for distributed memory applications, is implemented to model radiation heat transfer. Parallel scalability is analyzed for the coupled flow, chemistry, and radiation heat transfer processes. Weak and strong scaling studies are conducted on up to 15,000 computational ranks, revealing robust performance when flow cells exceed 200 per rank. Chemistry evaluations dominate the computational cost in large 3D simulations, accounting for approximately 40% of the total runtime, while flow processes contribute around 35%, and radiation solver contributions remain below 10% due to reduced evaluation frequencies. GPU accelerated chemistry evaluation, implemented with Zero-RK, demonstrates significant promise, achieving up to a 4x speedup for workloads exceeding 30,000 cells per GPU. However, diminishing returns are observed for smaller workloads due to CPU-GPU communication overhead. This study identifies key challenges, including memory bottlenecks and the effects of domain partitioning on flow scalability, while highlighting the potential of GPU-accelerated chemistry to reduce computational costs. In conclusion, these findings provide realizable run configurations for 2D, 3D, and GPU-accelerated cases, offering insights for optimizing reactive flow solvers.