Portable Programming Model Exploration for LArTPC Simulation in a Heterogeneous Computing Environment: OpenMP vs. SYCL
Explore the source record for details and available documents.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Engineering topics
Atif, Mohammad
Publications and source records attributed to Atif, Mohammad.
Direct numerical simulations of activation and deactivation in turbulent atmospheric clouds
Significant knowledge gaps remain in our understanding of turbulence–cloud–aerosol interactions in the Earth's atmosphere, and direct numerical simulation (DNS) has increasingly become an indispensable tool to fill such critical knowledge gaps. Here, this study is an extension of our previous DNS model [Gao et al., J. Geophys. Res.: Atmos., 123(4), 2194–2214 (2018)], with a focus on the activation of aerosol particles into cloud droplets and deactivation of cloud droplets into aerosol particles in a microscale cloud environment. The effects of turbulence intensity, particle curvature, and solute, as well as the initial distributions of the aerosol particles (monodisperse or polydisperse) are investigated. The governing equations for the flow of air, temperature, and water vapor mixing ratio are solved numerically in the Eulerian fashion, assuming homogeneous and isotropic turbulence. The dynamics of the aerosol and cloud particles are calculated with the Lagrangian particle tracking method. The results show that the deviations of the thermodynamic variables from their respective means are significantly reduced, the activation process appears to be delayed, and the deactivation process occurs more rapidly, as the turbulence intensity is increased. The inclusion of particle curvature and solute effects, as well as polydispersity, tends to retard the activation of aerosols into cloud droplets. It is also observed that fluctuations in supersaturation broaden the spread of particle radii, and the broadening is followed by a narrowing as turbulent homogenization reduces thermodynamic fluctuations over time.
54 ENVIRONMENTAL SCIENCES↗
Towards accelerating particle-resolved direct numerical simulation with neural operators
In this paper, we present our ongoing work aimed at accelerating a particle-resolved direct numerical simulation model designed to study aerosol–cloud–turbulence interactions. The dynamical model consists of two main components—a set of fluid dynamics equations for air velocity, temperature, and humidity, coupled with a set of equations for particle (i.e., cloud droplet) tracing. Rather than attempting to replace the original numerical solution method in its entirety with a machine learning (ML) method, we consider developing a hybrid approach. We exploit the potential of neural operator learning to yield fast and accurate surrogate models and, in this study, develop such surrogates for the velocity and vorticity fields. We discuss results from numerical experiments designed to assess the performance of ML architectures under consideration as well as their suitability for capturing the behavior of relevant dynamical systems.
54 ENVIRONMENTAL SCIENCES↗
A new portable random number generator wrapper library
Random number generator is an important component of many scientific projects. Many projects are written using programming models (like OpenMP and SYCL) to target different architectures. However, some programming models do not provide a random number generator. In this work, we introduce our random number generator wrapper. It is a header-only library that supports three distributions of random numbers: uniform, normal, and poisson. On the GPU backend, it wraps the cuRAND and rocRAND library, and supports various random number engines. It also wraps random123, a counterbased random number generator, on both CPU and GPU. With this library, we can generate random numbers with a few lines of code and target both GPU and multi-thread CPU with the same code. We also investigate the performance and scalability of this wrapper on different architectures with different engines and the number of cores.
97 MATHEMATICS AND COMPUTING↗
Porting ATLAS Fast Calorimeter Simulation to GPUs with Performance Portable Programming Models
FastCaloSim is a parameterized simulation of the particle energy response and of the energy distribution in the ATLAS calorimeter. It is a relatively small and self-contained package with massive inherent parallelism and captures the essence of GPU offloading via important operations like data transfer, memory initialization, floating point operations, and reduction. It was identified by the High Energy Physics Center for Computational Excellence project as a good testbed for evaluating the performance and ease of portability of programming models. In this paper, we will discuss the results of our evaluation of the porting process to Kokkos, SYCL, Alpaka, OpenMP and std::par (nvc++), and compare performance on NVIDIA, AMD and Intel GPUs, as well as multicore CPUs.
97 MATHEMATICS AND COMPUTING↗
Emulator of PR‐DNS: Accelerating Dynamical Fields With Neural Operators in Particle‐Resolved Direct Numerical Simulation
Abstract Particle‐resolved direct numerical simulations (PR‐DNS) play an increasing role in investigating aerosol‐cloud‐turbulence interactions at the most fundamental level of processes. However, the high computational cost associated with high resolution simulations poses considerable challenges for large domain or long duration simulation using PR‐DNS. To address these issues, here we present an emulator of the complex physics‐based PR‐DNS developed by use of the data‐driven Fourier Neural Operator (FNO) method. The effectiveness of the method is showcased by presenting turbulence and temperature fields in a two‐dimensional space. The results demonstrate high accuracy at various resolutions and the emulator is two orders of magnitude cheaper in terms of computational demand compared to the physics‐based PR‐DNS model. Furthermore, the FNO emulator exhibits strong generalization capabilities for different initial conditions and ultra‐high‐resolution without the need to retrain models. These findings highlight the potential of the FNO method as a promising tool to simulate complex fluid dynamics problems with high accuracy, computational efficiency, and generalization capabilities, enhancing our understanding of the aerosol‐cloud‐precipitation system.
99 GENERAL AND MISCELLANEOUS↗
Effect of outer-cylinder rotation on the radially heated Taylor–Couette flow
A Taylor–Couette setup with radial heating is considered where a Boussinesq fluid is sheared in the annular region between two concentric, independently rotating cylinders maintained at different temperatures. Linear stability analysis is performed to determine the Taylor number for the onset of instability. Two radius ratios corresponding to wide and thin gaps with several rotation rate ratios are considered. The rotation of the outer cylinder is found to have a general stabilizing effect on the stability threshold as compared to pure inner-cylinder rotation, with a few exceptions. The radial heating sets up an axial flow which breaks the reflection symmetry of isothermal Taylor–Couette flow in the axial coordinate. This symmetry breaking separates linear stability thresholds, and we find the fastest growing modes with both positive and negative azimuthal numbers for different parameters. Another important finding of the current study is the discovery of unstable modes in the Rayleigh-stable regime. Furthermore, closed disconnected neutral curves (CDNCs) are observed for both wide and thin gaps which can separate from or merge into open neutral–stability curves. Alternatively, CDNCs can also morph into open neutral stability curves as the rotation rate ratio is changed. CDNCs are observed to be sensitive to changes in control parameters, and their appearance/disappearance is shown to induce discontinuous jumps in the critical Taylor number. Finally, for both wide and thin gaps, the fastest-growing modes found in the pure corotation case are shown to have their origins in the instability islands at smaller values of rotation rate ratios.
97 MATHEMATICS AND COMPUTING↗
Evaluating Portable Parallelization Strategies for Heterogeneous Architectures in High Energy Physics
High-energy physics (HEP) experiments have developed millions of lines of code over decades that are optimized to run on traditional x86 CPU systems. However, we are seeing a rapidly increasing fraction of floating point computing power in leadership-class computing facilities and traditional data centers coming from new accelerator architectures, such as GPUs. HEP experiments are now faced with the untenable prospect of rewriting millions of lines of x86 CPU code, for the increasingly dominant architectures found in these computational accelerators. This task is made more challenging by the architecture-specific languages and APIs promoted by manufacturers such as NVIDIA, Intel and AMD. Producing multiple, architecture-specific implementations is not a viable scenario, given the available person power and code maintenance issues. The Portable Parallelization Strategies team of the HEP Center for Computational Excellence is investigating the use of Kokkos, SYCL, OpenMP, std::execution::parallel and alpaka as potential portability solutions that promise to execute on multiple architectures from the same source code, using representative use cases from major HEP experiments, including the DUNE experiment of the Long Baseline Neutrino Facility, and the ATLAS and CMS experiments of the Large Hadron Collider. This cross-cutting evaluation of portability solutions using real applications will help inform and guide the HEP community when choosing their software and hardware suites for the next generation of experimental frameworks. We present the outcomes of our studies, including performance metrics, porting challenges, API evaluations, and build system integration.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Portable Programming Model Exploration for LArTPC Simulation in a Heterogeneous Computing Environment: OpenMP vs. SYCL
The evolution of the computing landscape has resulted in the proliferation of diverse hardware architectures, with different flavors of GPUs and other compute accelerators becoming more widely available. To facilitate the efficient use of these architectures in a heterogeneous computing environment, several programming models are available to enable portability and performance across different computing systems, such as Kokkos, SYCL, OpenMP and others. As part of the High Energy Physics Center for Computational Excellence (HEP-CCE) project, we investigate if and how these different programming models may be suitable for experimental HEP workflows through a few representative use cases. One of such use cases is the Liquid Argon Time Projection Chamber (LArTPC) simulation which is essential for LArTPC detector design, validation and data analysis. Following up on our previous investigations of using Kokkos to port LArTPC simulation in the Wire-Cell Toolkit (WCT) to GPUs, we have explored OpenMP and SYCL as potential portable programming models for WCT, with the goal to make diverse computing resources accessible to the LArTPC simulations. In this work, we describe how we utilize relevant features of OpenMP and SYCL for the LArTPC simulation module in WCT. We also show performance benchmark results on multi-core CPUs, NVIDIA and AMD GPUs for both the OpenMP and the SYCL implementations. Comparisons with different compilers will also be given where appropriate.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Finding Hidden Patterns in High Resolution Wind Flow Model Simulations
Wind flow data is critical in terms of investment decisions and policy making. High resolution data from wind flow model simulations serve as a supplement to the limited resource of original wind flow data collection. Given the large size of data, finding hidden patterns in wind flow model simulations are critical for reducing the dimensionality of the analysis. In this work, we first perform dimension reduction with two autoencoder models: the CNN-based autoencoder (CNN-AE) [1], and hierarchical autoencoder (HIER-AE) [2], and compare their performance with the Principal Component Analysis (PCA). We then investigate the super-resolution of the wind flow data. By training a Generative Adversarial Network (GAN) with 300 epochs, we obtained a trained model with 2× resolution enhancement. We compare the results of GAN with Convolutional Neural Network (CNN), and GAN results show finer structure as expected in the data field images. Also, the kinetic energy spectra comparisons show that GAN outperforms CNN in terms of reproducing the physical properties for high wavenumbers and is critical for analysis where high-wavenumber kinetics play an important role.
97 MATHEMATICS AND COMPUTING↗