Search NASA⌕ Search

Engineering topics

Yeung, P. K.

Publications and source records attributed to Yeung, P. K..

GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading

Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first exascale computer, Frontier, comprising AMD MI250x GPUs with HPE's Slingshot interconnect and operated by the US Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCm library calls. With 99% of computing power on Frontier being on the GPU, leaving the CPU idle leads to a net performance gain via avoiding the overhead of data movement between host and device except when needed for some I/O purposes. Memory footprint including the size of communication buffers for MPI_ALLTOALL is managed carefully to maximize the largest problem size possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from 2048 3 on a single node to 32768 3 on 4096 or 8192 nodes out of 9408 on the system. Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are implemented. The present code suite (labeled by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance attained is highly favorable in both weak scaling and strong scaling, with notable departures only for 2048 3 where communication is entirely intra-node, and for 32768 3 , where a challenge due to small message sizes does arise. Communication performance is addressed further using a lightweight test code that performs all-to-all communication in a manner matching the full turbulence simulation code. Performance at large problem sizes is affected by both small message size due to high node counts as well as dragonfly network topology features on the machine, but is consistent with official expectations of sustained performance on Frontier. Overall, although not perfect, the scalability achieved at the extreme problem size of 32768 3 (and up to 8192 nodes — which corresponds to hardware rated at just under 1 exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled by this code and its extensions are to be reported separately in the near future.

3D fast Fourier transform↗

OpenMP application experiences: Porting to accelerated nodes

As recent enhancements to the OpenMP specification become available in its implementations, there is a need to share the results of experimentation in order to better understand the OpenMP implementation’s behavior in practice, to identify pitfalls, and to learn how the implementations can be effectively deployed in scientific codes. We report on experiences gained and practices adopted when using OpenMP to port a variety of ECP applications, mini-apps and libraries based on different computational motifs to accelerator-based leadership-class high-performance supercomputer systems at the United States Department of Energy. Additionally, we identify important challenges and open problems related to the deployment of OpenMP. Through our report of experiences, we find that OpenMP implementations are successful on current supercomputing platforms and that OpenMP is a promising programming model to use for applications to be run on emerging and future platforms with accelerated nodes.

97 MATHEMATICS AND COMPUTING↗

Steep Cliffs and Saturated Exponents in Three-Dimensional Scalar Turbulence

Herein the intermittency of a passive scalar advected by three-dimensional Navier-Stokes turbulence at a Taylor-scale Reynolds number of 650 is studied using direct numerical simulations on a 4096 3 grid; the Schmidt number is unity. By measuring scalar increment moments of high orders, while ensuring statistical convergence, we provide unambiguous evidence that the scaling exponents saturate to 1.2 for moment orders beyond about 12, indicating that scalar intermittency is dominated by the most singular shocklike cliffs in the scalar field. We show that the fractal dimension of the spatial support of steep cliffs is about 1.8, whose sum with the saturation exponent value of 1.2 adds up to the space dimension of 3, thus demonstrating a deep connection between the geometry and statistics in turbulent scalar mixing. The anomaly for the fourth and sixth order moments is comparable to that in the Kraichnan model for the roughness exponent of 4/3.

42 ENGINEERING↗

Structure functions and applicability of Yaglom's relation in passive-scalar turbulent mixing at low Schmidt numbers with uniform mean gradient

An extensive direct numerical simulation database over a wide range of Reynolds and Schmidt numbers is used to examine the Schmidt number dependence of the structure function of passive scalars and the applicability of the so-called Yaglom's relation in isotropic turbulence with a uniform mean scalar gradient. For the moderate Reynolds numbers available, the limited range of scales in scalar fields of very low Schmidt numbers (as low as 1/2048) is seen to lead to weaker intermittency, and weaker alignment between velocity gradients and principal strain rates. Strong departures from both Obukhov-Corrsin scaling for second-order structure functions and Yaglom's relation for the mixed velocity-scalar third-order structure function are observed. Finally, evaluation of different terms in the scalar structure function budget equation assuming statistical stationarity in time shows that, if the Schmidt number is very low, at intermediate scales production and diffusion terms (instead of advection) are major contributors in the balance against dissipation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Gaussian Lagrangian stochastic models for multi-particle dispersion

We have extended the “well-mixed” two-particle stochastic models for 3D Gaussian turbulence to n particles, and have performed calculations for clusters of n ≤ 6 particles. The particle joint motions are Gaussian and are constrained by pair-wise spatial correlations. This neglects non-Gaussian properties of the two-point velocity distribution and also neglects multi-point correlations. It also takes no account of intermittency. Although the models do not predict the growth of the separation of particles in the cluster satisfactorily, we find that they do give a good representation of the shape statistics for the cluster in comparison with direct numerical simulation results. We conclude that the pair-wise spatial structure of the turbulence accounts for most of the observed characteristics of the shape of multi-particle clusters in turbulence, and that non-Gaussian and multi-point features of the turbulence are of secondary importance.

42 ENGINEERING↗

Spectrum of passive scalars of high molecular diffusivity in turbulent mixing

Abstract We consider the mixing of passive scalars transported in turbulent flow, with a molecular diffusivity that is large compared to the kinematic viscosity of the fluid. This particular case of mixing has not received much attention in experiment or simulation even though the first putative theory, due to Batchelor, Howells & Townsend ( J. Fluid Mech. , vol. 5, 1959, pp. 134–139), is now more than 50 years old. We study the problem using direct numerical simulation of decaying scalar fields in steadily sustained homogeneous turbulence as the Schmidt number (the ratio of the kinematic viscosity of the fluid to the molecular diffusivity of the scalar) is allowed to vary from $1/ 8$ to $1/ 2048$ for two values of the microscale Reynolds number, ${R}_{\lambda } \approx 140$ and $\approx $ 240. The simulations show that the passive scalar spectrum assumes a slope of $- 17/ 3$ in a range of scales, as predicted by the theory, when the Schmidt number is small and the Reynolds number is simultaneously large. The observed agreement between theory and simulation in the prefactor in the spectrum is not perfect. We assess the reasons for this discrepancy by a careful examination of the scalar evolution equation in the light of the assumptions of the theory, and conclude that the finite range of scales resolved in simulations is the main reason. Numerical issues specific to the regime of very low Schmidt numbers are also addressed briefly.

Mechanics↗

Dissipation, enstrophy and pressure statistics in turbulence simulations at high Reynolds numbers

Abstract We use data from well-resolved direct numerical simulations at Taylor-scale Reynolds numbers from 140 to 1000 to study the statistics of energy dissipation rate and enstrophy density (i.e. the square of local vorticity). Despite substantial variability in each of these variables, their extreme events not only scale in a similar manner but also progressively tend to occur spatially together as the Reynolds number increases. Though they possess non-Gaussian tails of enormous amplitudes, ratios of some characteristic properties can be closely linked to those of isotropic Gaussian random fields. We present results also on statistics of the pressure Laplacian and conditional mean pressure given both dissipation and enstrophy. At low Reynolds number intense negative pressure fluctuations are preferentially associated with rotation-dominated regions but at high Reynolds number both high dissipation and high enstrophy have similar effects.

Mechanics↗

Some results on the Reynolds number scaling of pressure statistics in isotropic turbulence

Using data from direct numerical simulations in the Reynolds number range , where is the Taylor microscale Reynolds number, we assess the Reynolds number scaling of the microscale and the integral length scale of pressure fluctuations in homogeneous and isotropic turbulence. The root-mean-square (rms) pressure (in kinematic units) is about , where is the rms velocity in any one direction. The ratio of the pressure microscale to the (longitudinal) velocity Taylor microscale is a constant of about 0.74 for very low Reynolds numbers but increases approximately as at high Reynolds numbers. We discuss these results in the context of the existing theory and provide plausible explanations, based on intermittency, for their observed trends.

Mathematics↗

Kolmogorov similarity scaling for one-particle Lagrangian statistics

Here, we use direct numerical simulation data up to a Taylor scale Reynolds number R λ =1000 to investigate Kolmogorov similarity scaling in the inertial sub-range for one-particle Lagrangian statistics. While similarity scaling is not achieved at these Reynolds numbers for the Lagrangian velocity structure function, we show clearly that it is achieved for the Lagrangian acceleration frequency spectrum and the scaling range becomes wider with increasing Reynolds number. Stochastic and heuristic model calculations suggest that the difference in behavior observed for the structure function and spectrum is simply a consequence of different rates of convergence to scaling behavior with increasing Reynolds number. Our estimate C 0 ≈6.9±0.2 for the Lagrangian structure function constant is close to earlier estimates based on extrapolation of the peak value of the compensated structure function. The findings presented here suggest prospects for studying Kolmogorov similarity for Lagrangian statistics using the latest innovations in simulation, and measurement techniques are more hopeful than previously suggested in the literature.

42 ENGINEERING↗

Multi-particle and tetrad statistics in numerical simulations of turbulent relative dispersion

The evolution in size and shape of three and four-particle clusters (triangles and tetrads, respectively) in isotropic turbulence is studied using direct numerical simulations at grid resolution up to 4096 3 and Taylor-scale Reynolds numbers from 140 to 1000. One main issue is the attainment of inertial range behavior at high Reynolds number, while the small- and large-time limits of ballistic and diffusive regimes, respectively, are also considered in some detail. Tetrad size expressed by the volume (V) and (more appropriately) the gyration radius (R) is demonstrated to display inertial range scaling consistent with a Richardson constant close to 0.56 for two-particle relative dispersion. For tetrads of initial size in a suitable range moments of shape parameters, including the ratio V 2/3 /R 2 and normalized eigenvalues of a moment-of-inertia-like dispersion tensor, show a regime of near-constancy which is identified with inertial-range scaling. Sheet-like structures are dominant in this period, while pancakes and needles are more prevalent at later times. For triangles taken from different faces of each tetrad effects of the initial shape (isosceles right-angled or equilateral) are retained only for about one Batchelor time scale. In the inertial range there is a prevalence of nearly isosceles triangles of two long sides and one short side, representing one particle moving away from the other two which are still close together. In general, measures of shape display asymptotic scaling ranges more readily than measures of size. With some caveats, the simulation results are also compared with the limited literature available for multiparticle cluster dispersion in turbulent flow.

42 ENGINEERING↗

Numerical Study of Rotating Turbulence with External Forcing

Direct numerical simulation at 256(exp 3) resolution have been carried out to study the response of isotropic turbulence to the concurrent effects of solid-body rotation and numerical forcing at the large scales. Because energy transfer to the smaller scales is weakened by rotation, energy input from forcing gradually builds up at the large scales, causing the overall kinetic energy to increase. At intermediate wavenumbers the energy spectrum undergoes a transition from a limited k(exp -5/3) inertial range to k(exp -2) scaling recently predicted in the literature. Although the Reynolds stress tensor remains approximately isotropic and three-components, evidence for anisotropy and quasi- two-dimensionality in length scales and spectra in different velocity components and directions is strong. The small scales are found to deviate from local isotropy, primarily as a result of anisotropic transfer to the high wavenumbers. To understand the spectral dynamics of this flow we study the detailed behavior of nonlinear triadic interactions in wavenumber space. Spectral transfer in the velocity component parallel to the axis of rotation is qualitatively similar to that in non-rotating turbulence; however the perpendicular component is characterized by a greatly suppressed energy cascade at high wavenumber and a local reverse transfer at the largest scales. The broader implications of this work are briefly addressed.

Yeung, P. K.↗

On the Universality of the Kolmogorov Constant in Numerical Simulations of Turbulence

Motivated by a recent survey of experimental data, we examine data on the Kolmogorov spectrum constant in numerical simulations of isotropic turbulence, using results both from previous studies and from new direct numerical simulations over a range of Reynolds numbers (up to 240 on the Taylor scale) at grid resolutions up to 512(exp 3). It is noted that in addition to k(exp -5/3) scaling, identification of a true inertial range requires spectral isotropy in the same wavenumber range. We found that a plateau in the compensated three-dimensional energy spectrum at k(eta) approx. = 0.1 - -0.2, commonly used to infer the Kolmogorov constant from the compensated three-dimensional energy spectrum, actually does not represent proper inertial range behavior. Rather, a proper, if still approximate, inertial range emerges at k(eta) approx. = 0.02 - 0.05 when R(sub lambda) increases beyond 140. The new simulations indicate proportionality constants C(sub 1) and C in the one- and three-dimensional energy spectra respectively about 0.60 and 1.62. If the turbulence were perfectly isotropic then use of isotropy relations in wavenumber space (C(sub 1) = 18/55 C) would imply that C(sub 1) approx. = 0.53 for C = 1.62, in excellent agreement with experiments. However the one- and three-dimensional estimates are not fully consistent, because of departures (due to numerical and statistical limitations) from isotropy of the computed spectra at low wavenumbers. The inertial scaling of structure functions in physical space is briefly addressed. Since DNS is still restricted to moderate Reynolds numbers, an accurate evaluation of the Kolmogorov constant is very difficult. We focus on providing new insights on the interpretation of Kolmogorov 1941 similarity in the DNS literature and do not consider issues pertaining to the refined similarity hypotheses of Kolmogorov (K62).

Yeung, P. K.↗

Analysis of Subgrid Eddy Diffusivity in Turbulence Using Direct Numerical Simulation Databases

Direct numerical simulation data in isotropic turbulence at Taylor-scale Reynolds number 90 are used to compute subgrid transfer and spectral eddy diffusivities for passive scalars at Schmidt numbers 1 and 1/8. The results are consistent with spectral transfer processes dominated by a local forward cascade in wavenumber space which is a consequence of moderately nonlocal interactions in the scalar fields. Data decompositions show that resolvable-subgrid interactions make (in general) substantially stronger contributions than fully-subgrid interactions to the eddy diffusivity, which has a non-negligible inverse cascade, or backscatter, component. The eddy diffusivity has a strong cusp at wavenumbers close to the cutoff value separating resolvable and subgrid ranges. At low wavenumbers the eddy diffusivity is generally small, but there the details depend on Schmidt number. The behavior of the subgrid turbulent Schmidt number (the ratio of effective eddy viscosity to eddy diffusivity) is observed to depend strongly on Schmidt number. In interpreting the results at a given spectral cutoff it is important to note that the spectrum of less diffusive scalars (of higher Schmidt number) has more high-wavenumber content.

Yeung, P. K.↗

Analysis of subgrid models using direct and large-eddy simulations of isotropic turbulence

Direct and large eddy simulations of forced and decaying isotropic turbulence have been performed using a pseudospectral and a finite-difference code. Subgrid models that include a one-equation subgrid kinetic energy model with and without a stochastic backscatter forcing term and a new scale similarity model have been analyzed in both Fourier space and physical space. The Fourier space analysis showed that the energy transfer across the cutoff wavenumber k(sub c) is dominated by local interaction. The correlation between the exact and the modeled (by a spectral eddy viscosity) nonlinear terms and the subgrid energy transfer in physical space was found to be quite low. In physical space, a similar correlation analysis was carried out using top hat filtering. Results show that the subgrid stress and the energy flux predicted by the subgrid models correlates very well with the exact data. The scale similarity model showed very high correlation for reasonable grid resolution. However, with decrease in grid resolution, the scale similarity model became more uncorrelated, when compared to the kinetic energy subgrid model. The subgrid models were then used for large-eddy simulations for a range of Reynolds number. It was determined that the dissipation was modeled poorly and that the correlation with the exact results was quite low for all the models. In general, for coarse grid resolution, the scale similarity model consistently showed very low correlation while the kinetic energy model showed a relatively higher correlation. These results suggest that to use the scale similarity model relatively fine grid resolution may be required, whereas, the kinetic energy model could be used even in coarse grid.

Menon, S.↗

Scale disparity and spectral transfer in anisotropic numerical turbulence

To study the effect of cancellations within long-range interactions on local isotropy at the small scales, we calculate explicitly the degree of cancellation in distant interactions in the simulations of Yeung & Brasseur and Yeung, Brasseur & Wang using the single scale disparity parameter 's' developed by Zhou. In the simulations, initially isotropic simulated turbulence was subjected to coherent anisotropic forcing at the large scales and the smallest scales were found to become anisotropic as a consequence of direct large-small scale couplings. We find that the marginally distant interactions in the simulation do not cancel out under summation and that the development of small-scale anisotropy is indeed a direct consequence of the distant triadic group, as argued by Yeung, et. al. A reduction of anisotropy at later times occurs as a result of the isotropizing influences of more local energy-cascading triadic interactions. Nevertheless, the local-to-nonlocal triadic group persists as an isotropizing influence at later times. We find that, whereas long-range interactions, in general, contribute little to net energy transfer into or out of a high wavenumber shell k, the anisotropic transfer of component energy within the shell increases with increasing scale separations. These results are consistent with results by Zhou, and Brasseur & Wei, and suggest that the anisotropizing influences of long range interactions should persist to higher Reynolds numbers. The residual effect of the forced distant group in this low-Reynolds number simulation is found to be forward cascading, on average.

Zhou, YE↗