Search NASA⌕ Search

SEARCH · Search NASA

Results for “exascale simulation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

In Situ Inference for Earth System Predictability

An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗

Large-Scale Computational Fluid Dynamics Simulations of Aerospace Configurations on the Frontier Exascale System

Over the past fifteen years, the high performance computing landscape has undergone a seismic shift in both hardware and software paradigms, which has been necessary to realize a 1000× leap in computational performance while meeting stringent constraints on power consumption. A historical overview of a long-term research effort aimed at addressing these challenges within the context of a commonly-used aerospace computational fluid dynamics (CFD) application is presented. Details of the current implementation as they relate to the new era of exascale-relevant hardware architectures and programming models are described. Two large-scale simulations of aerospace configurations are performed using the entire Frontier exascale system, currently ranked as the most powerful supercomputing system in the world. The effort serves to address a 2024 milestone posed a decade ago by the seminal CFD Vision 2030 Study.

Eric J Nielsen↗

Subseasonal Tropical Convection Characteristics in the Energy Exascale Earth System Model Version 2

Accurate simulation of subseasonal tropical moist convection remains a key challenge for Earth system models. The difficulties stem from the reliance of cumulus cloud processes on model parameterizations and the need to represent the multiscale nature of interactions among clouds, radiation, moisture, circulation, and surface energy fluxes. Equatorial convection drives circulation anomalies that can affect weather patterns and extremes globally, motivating efforts to better understand and simulate these tropical disturbances. Here, a detailed review of subseasonal tropical convective behavior as simulated in the Energy Exascale Earth System Model version 2 (E3SMv2) is presented, with comparison to its predecessor version 1 (E3SMv1) and reference data sets. Model structural changes to the deep convective trigger function and surface fluxes, along with parametric tuning of the cloud and microphysics schemes, together result in an improved depiction of organized tropical convection across scales. In particular, E3SMv2 exhibits a more realistic Madden‐Julian oscillation (MJO) and low‐frequency Kelvin waves—owing to a sharper time mean equatorial meridional moisture gradient and improved convection‐circulation coupling —as well as a better depiction of MJO Northern Hemisphere teleconnections. Despite these improvements, subseasonal precipitation variance continues to be strongly underestimated in E3SMv2. Use of a cloud plume model also reveals that the coupling between daily averaged tropical precipitation and lower tropospheric instability in E3SM is inconsistent with observations, a bias that could potentially impact the simulation of intraseasonal disturbances.

54 ENVIRONMENTAL SCIENCES↗

Advancing Quantum Many-Body GW Calculations on Exascale Supercomputing Platforms

Advanced ab initio materials simulations face growing challenges as increasing systems and phenomena complexity requires higher accuracy, driving up computational demands. Quantum many-body GW methods are state-of-the-art for treating electronic excited states and couplings but often hindered due to the costly numerical complexity. Here, we present innovative implementations of advanced GW methods within the BerkeleyGW package, enabling large-scale simulations on Frontier and Aurora exascale platforms. Our approach demonstrates exceptional versatility for complex heterogeneous systems with up to 17,574 atoms, along with achieving true performance portability across GPU architectures. We demonstrate excellent strong and weak scaling to thousands of nodes, reaching double-precision core-kernel performance of 1.069 ExaFLOP/s on Frontier (9,408 nodes) and 707.52 PetaFLOP/s on Aurora (9,600 nodes), corresponding to 59.45% and 48.79% of peak, respectively. Our work demonstrates a breakthrough in utilizing exascale computing for quantum materials simulations, delivering unprecedented predictive capabilities for rational designs of future quantum technologies.

Zhang, Benran [University of Southern California, ↗

Towards Ab-Initio Simulations of Crystalline Defects at the Exascale Using Spectral Quadrature Density Functional Theory

Defects in crystalline solids play a crucial role in determining properties of materials at the nano, meso- and macroscales, such as the coalescence of vacancies at the nanoscale to form voids and prismatic dislocation loops or diffusion and segregation of solutes to nucleate precipitates, phase transitions in magnetic materials via disorder and doping. First principles Density Functional Theory (DFT) simulations can provide a detailed understanding of these phenomena. However, the number of atoms needed to correctly simulate these systems is often beyond the reach of many widely used DFT codes. The aim of this article is to discuss recent advances in first principles modeling of crystal defects using the spectral quadrature method. The spectral quadrature method is linear scaling with respect to the number of atoms, permits spatial coarse-graining, and is capable of simulating non-periodic systems embedded in a bulk environment, which allows the application of appropriate boundary conditions for simulations of crystalline defects. In this article, we discuss the state-of-the-art in ab-initio modeling of large metallic systems of the order of several thousand atoms that are suitable for utilizing exascale computing resourses.

36 MATERIALS SCIENCE↗

ExaAM: Metal additive manufacturing simulation at the fidelity of the microstructure

Additive manufacturing (AM), or 3D printing, of metals is transforming the fabrication of components, in part by dramatically expanding the design space, allowing optimization of shape and topology. However, although the physical processes involved in AM are similar to those of welding, a field with decades of experimental, modeling, simulation, and characterization experience, qualification of AM parts remains a challenge. The availability of exascale computational systems, particularly when combined with data-driven approaches such as machine learning, enables topology and shape optimization as well as accelerated qualification by providing process-aware, locally accurate microstructure and mechanical property models. We describe the physics components comprising the Exascale Additive Manufacturing simulation environment and report progress using highly resolved melt pool simulations to inform part-scale finite element thermomechanics simulations, drive microstructure evolution, and determine constitutive mechanical property relationships based on those microstructures using polycrystal plasticity. We report on implementation of these components for exascale computing architectures, as well as the multi-stage simulation workflow that provides a unique high-fidelity model of process–structure–property relationships for AM parts. In addition, we discuss verification and validation through collaboration with efforts such as AM-Bench, a set of benchmark test problems under development by a team led by the National Institute of Standards and Technology.

3D printing↗

GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading

Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first exascale computer, Frontier, comprising AMD MI250x GPUs with HPE's Slingshot interconnect and operated by the US Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCm library calls. With 99% of computing power on Frontier being on the GPU, leaving the CPU idle leads to a net performance gain via avoiding the overhead of data movement between host and device except when needed for some I/O purposes. Memory footprint including the size of communication buffers for MPI_ALLTOALL is managed carefully to maximize the largest problem size possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from 2048 3 on a single node to 32768 3 on 4096 or 8192 nodes out of 9408 on the system. Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are implemented. The present code suite (labeled by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance attained is highly favorable in both weak scaling and strong scaling, with notable departures only for 2048 3 where communication is entirely intra-node, and for 32768 3 , where a challenge due to small message sizes does arise. Communication performance is addressed further using a lightweight test code that performs all-to-all communication in a manner matching the full turbulence simulation code. Performance at large problem sizes is affected by both small message size due to high node counts as well as dragonfly network topology features on the machine, but is consistent with official expectations of sustained performance on Frontier. Overall, although not perfect, the scalability achieved at the extreme problem size of 32768 3 (and up to 8192 nodes — which corresponds to hardware rated at just under 1 exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled by this code and its extensions are to be reported separately in the near future.

3D fast Fourier transform↗

Towards Ultra-high-resolution E3SM Land Modeling on Exascale Computers

Here we present an ultra-high-resolution E3SM land model (uELM) for high-fidelity land simulations targeting new Exascale computers. After considering modeling infrastructure compatibility and ELM software features, we designed a parallel model for the uELM development targeting hybrid architectures of new US Exascale computers. We also described a function unit test framework to expedite the piece-wise code porting (with compiler directives), verification, and global variable management. Furthermore, in this study, we report an early uELM model development using OpenACC within a function unit test framework on a pre-Exascale computer, demonstrate the performance of a uLEM submodel with a 3.0-time speedup, and summarize the code porting experience regarding global variable handling, deepcopy, memory reduction, and parallel loop reconstruction.

97 MATHEMATICS AND COMPUTING↗

Adaptive Mesh Refinement Simulations for Turbulent Reacting Flow

With the increased availability of exascale computing hardware, detailed simulations of realistic devices can be performed at practically relevant time and length scales. Insights into the multiscale driving mechanisms in compressible reacting flow systems with complex geometry, such as combustors, can be used for design optimization and technology improvements. However, to effectively perform these simulations, advanced numerical algorithms must be used to maintain solution accuracy without incurring undue computational costs. PeleC, part of the Pele suite of codes, leverages block-structured adaptive mesh refinement (AMR) through the AMReX library to capture fine-scale flow features in compressible reacting flows. In this talk, we discuss recent improvements to the numerical algorithms, particularly in regard to describing flows at complex boundary structures, and PeleC's performance on exascale computing hardware. We will demonstrate that PeleC is well-suited for modern, extreme-scale, heterogenous compute platforms.

combustion↗

Mesoscale Convective Systems in a Superparameterized E3SM Simulation at High Resolution

Accurately representing mesoscale convective systems (MCSs) is crucial to simulating the energy and water cycles in global climate models. Using a novel MCS identification and tracking algorithm applied to observations and model simulations, we evaluate how well the Energy Exascale Earth System Model simulates MCSs over the central US. Simulations performed by E3SM at 25 km grid spacing with (SP-E3SM) and without (E3SM) superparameterization are compared and evaluated against observations using multiple metrics highlighting important MCS characteristics. Compared to E3SM, SP-E3SM better simulates MCS number and MCS precipitation amount, diurnal cycle, propagation, and the probability distribution of precipitation rate in both spring and summer. The improvement from SP is partly contributed by improvement in simulating the large-scale environments, featuring enhanced atmospheric low-level moisture and larger moisture transport to the central US relative to E3SM. However, SP-E3SM still underestimates MCS precipitation amount, particularly in summer. This underestimation is closely related to the negative bias in MCS precipitation intensity, although the drier environments simulated during summer also contributes to underestimation of MCS frequency. Without SP, the larger bias in MCS precipitation amount is closely related to the negative bias in MCS frequency, while the substantial dry bias in the large-scale environments also contributes to underestimation of MCS intensity. Our results suggest that SP improves MCS simulation by improving modeling of the large-scale environments and convection initiation, which are both major limiting factors in E3SM even at 25 km grid spacing where deep convection is represented by a cumulus parameterization.

54 ENVIRONMENTAL SCIENCES↗

Initial full core SMR simulations with NekRS

This document describes the completion of a recent milestone of the ExaSMR program concerning "Full core simulations". The ExaSMR project is developing tools for coupling Monte Carlo (MC) radiation transport solvers to a computational fluid dynamics (CFD) solver. Work is based on the Shift MC, OpenMC MC, and Nek5000/NekRS CFD codes. As part of this milestone, a novel set of pin-resolved CFD full-core simulations have been performed for the first time. These simulations represent a significant increase in capability in what is now possible with CFD on pre-Exascale systems. The simulations have been performed with the spectral element solvers NekRS on the supercomputer Summit. Multiple simulations campaigns have been conducted: (1) LES simulations in bare bundle, (2) RANS simulations in bare bundles, (3) RANS simulations with momentum sources and (4) Conjugate heat transfer calculations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

GPU-accelerated DNS of compressible turbulent flows

Here, this paper explores strategies to transform an existing CPU-based high-performance computational fluid dynamics solver, HyPar, for compressible flow simulations on emerging exascale heterogeneous (CPU+GPU) computing platforms. The scientific motivation for developing a GPU-enhanced version of HyPar is to simulate canonical turbulent flows at the highest resolution possible on such platforms. We show that optimizing memory operations and thread blocks results in 200x speedup of computationally intensive kernels compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication, we demonstrate both strong and weak scaling of our GPU-based HyPar implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of homogeneous isotropic turbulence in a triply periodic box on grids with up to 1024 3 points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We compare the wall times for CPU-only and CPU+GPU simulations. The results presented in the paper are obtained on the Summit and Lassen supercomputers at Oak Ridge and Lawrence Livermore National Laboratories, respectively.

97 MATHEMATICS AND COMPUTING↗

ExaWind: Predictive Wind Energy Simulations

This presentation describes the ExaWind project and the team's progress in creating a suite of performance-portable codes designed for predictive simulations of wind farms on next-generation exascale-class supercomputers. Such simulations will require the resolution of scales spanning many orders of magnitude, from blade boundary layers to wind farm flow structures. In the U.S., the first exascale systems will be GPU accelerated, and different GPU manufacturers have been chosen for the different systems. At the heart of the ExaWind software is a hybrid-solver approach based on the codes Nalu-Wind and AMR-Wind, which are computational fluid dynamics solvers for the incompressible Navier-Stokes equations. Nalu-Wind is an unstructured-grid code used to resolve wind turbine geometry and blade boundary layers, whereas AMR-Wind is a structured-grid background solver for atmospheric turbulent flow and turbine wake propagation. The models are coupled with overset meshes and global linear systems are approximated through a loose-coupling algorithm. Results will include validation-quality high-fidelity simulations and strong/weak scaling results from the Summit supercomputer.

49 EE - Wind and Water Power Program - Wind (EE-4W↗

Cholla-MHD: An Exascale-capable Magnetohydrodynamic Extension to the Cholla Astrophysical Simulation Code

Abstract We present an extension of the massively parallel, GPU native, astrophysical hydrodynamics code Cholla to magnetohydrodynamics (MHD). Cholla solves the ideal MHD equations in their Eulerian form on a static Cartesian mesh utilizing the Van Leer + constrained transport integrator, the HLLD Riemann solver, and reconstruction methods at second and third order. Cholla’s MHD module can perform ≈260 million cell updates per GPU-second on an NVIDIA A100 while using the HLLD Riemann solver and second order reconstruction. The inherently parallel nature of GPUs combined with increased memory in new hardware allows Cholla’s MHD module to perform simulations with resolutions ∼500 3 cells on a single high-end GPU (e.g., an NVIDIA A100 with 80 GB of memory). We employ GPU direct Message Passing Interface to attain excellent weak scaling on the exascale supercomputer Frontier, while using 74,088 GPUs and simulating a total grid size of over 7.2 trillion cells. A suite of test problems highlights the accuracy of Cholla’s MHD module and demonstrates that zero magnetic divergence in solutions is maintained to round off error. We also present new testing and CI tools using GoogleTest, GitHub Actions, and Jenkins that have made development more robust and accurate and ensure reliability in the future.

Astronomy & Astrophysics↗

Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package, version 2: assessing aerosols, clouds, and aerosol–cloud interactions via field campaign and long-term observations

Abstract. Poor representations of aerosols, clouds, and aerosol–cloud interactions (ACIs) in Earth system models (ESMs) have long been the largest uncertainties in predicting global climate change. Huge efforts have been made to improve the representation of these processes in ESMs, and the key to these efforts is the evaluation of ESM simulations with observations. Most well-established ESM diagnostics packages focus on the climatological features; however, they lack process-level understanding and representations of aerosols, clouds, and ACIs. In this study, we developed the Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package to facilitate the routine evaluation of aerosols, clouds, and ACIs simulated the Energy Exascale Earth System Model (E3SM) from the US Department of Energy (DOE). This paper documents its version 2 functionality (ESMAC Diags v2), which has substantial updates compared with version 1 (Tang et al., 2022a). The simulated aerosol and cloud properties have been extensively compared with in situ and remote-sensing measurements from aircraft, ship, surface, and satellite platforms in ESMAC Diags v2. It currently includes six field campaigns and two permanent sites covering four geographical regions: the eastern North Atlantic, the central US, the northeastern Pacific, and the Southern Ocean. These regions produce frequent liquid- or mixed-phase clouds, with extensive measurements available from the DOE Atmospheric Radiation Measurement user facility and other agencies. ESMAC Diags v2 generates various types of single-variable and multivariable diagnostics, including percentiles, histograms, joint histograms, and heatmaps, to evaluate the model representation of aerosols, clouds, and ACIs. Select examples highlighting the capabilities of ESMAC Diags are shown using E3SM version 2 (E3SMv2). In general, E3SMv2 can reasonably reproduce many observed aerosol and cloud properties, with biases in some variables such as aerosol particle and cloud droplet sizes and number concentrations. The coupling of aerosol and cloud number concentrations may be too strong in E3SMv2, possibly indicating a bias in processes that control aerosol activation. Furthermore, the liquid water path response to a perturbed cloud droplet number concentration behaves differently in E3SMv2 and observations, which warrants further study to improve the cloud microphysics parameterizations in E3SMv2.

54 ENVIRONMENTAL SCIENCES↗

Causal Interactions between Southern Ocean Polynyas and High-Latitude Atmosphere–Ocean Variability

Abstract Weddell Sea open-ocean polynyas have been observed to occasionally release heat from the deep ocean to the atmosphere, indicating that their sporadic appearances may be an important feature of high-latitude atmosphere–ocean variability. Yet, observations of the phenomenon are sparse and many standard-resolution models represent these features poorly, if at all. We use a fully coupled, synoptic-scale preindustrial control simulation of the Energy Exascale Earth System Model (E3SMv0-HR) to effectively simulate open-ocean polynyas and investigate their role in the climate system. Our approach employs statistical tests of Granger causality to diagnose local and remote drivers of, and responses to, polynya heat loss on interannual to decadal time scales. First, we find that polynya heat loss Granger causes a persistent increase in surface air temperature over the Weddell Sea, strengthening the local cyclonic wind circulation. Along with responding to polynyas, atmospheric conditions also facilitate their development. When the Southern Ocean experiences a rapid poleward shift in the circumpolar westerlies following a prolonged negative phase of the southern annular mode (SAM), Weddell Sea salinity increases, promoting density destratification and convection in the water column. Finally, we find that the reduction of surface heat fluxes during periods of full ice cover is not fully compensated by ocean heat transport into the high latitudes. This imbalance leads to a buildup of ocean heat content that supplies polynya heat loss. These results disentangle the complex, coupled climate processes that both enable the polynya’s existence and respond to it, providing insights to improve the representation of these highly episodic sea ice features in climate models.

Kaufman, Zachary S.↗

Making Uintah Performance Portable for Department of Energy Exascale Testbeds

To help ease ports to forthcoming Department of Energy (DOE) exascale systems, testbeds have been made available to select users. These testbeds are helpful for preparing codes to run on the same hardware and similar software as in their respective exascale systems. This paper describes how the Uintah Computational Framework, an open-source asynchronous many-task (AMT) runtime system, has been modified to be performance portable across the DOE Crusher, DOE Polaris, and DOE Sunspot testbeds in preparation for portable simulations across the exascale DOE Frontier and DOE Aurora systems. The Crusher, Polaris, and Sunspot testbeds feature the AMD MI250X, NVIDIA A100, and Intel PVC GPUs, respectively. This performance portability has been made possible by extending Uintah’s intermediate portability layer [18] to additionally support the Kokkos::HIP, Kokkos::OpenMPTarget, and Kokkos::SYCL back-ends. This paper also describes notable updates to Uintah’s support for Kokkos, which were required to make this extension possible. Results are shown for a challenging radiative heat transfer calculation, central to the University of Utah’s predictive boiler simulations. These results demonstrate single-source portability across AMD-, NVIDIA-, and Intel-based GPUs using various Kokkos back-ends.

Holmen, John↗