Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,693 records · Page 94
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Magnetic reconnection and thermal equilibration

When a magnetic field is forced to evolve on a time scale sev, as by footpoint motions driving the solar corona or non-axisymmetric instabilities in tokamaks, the magnetic field lines undergo large-scale changes in topology on a time scale approximately an order of magnitude longer than $\tau$ ev . But the physics that allows such changes operates on a time scale eight or more orders of magnitude slower. An analogous phenomenon occurs in air. Temperature equilibration occurs on a time scale approximately an order of magnitude longer than it takes air to cross a room, $\tau$ ev , although the physical mechanism that allows temperature equilibration is approximately four orders of magnitude slower than $\tau$ ev . The use of Lagrangian coordinates allows the fundamental equations to be solved and both phenomena explained. The theories of thermal equilibration and magnetic reconnection are developed in parallel to help readers obtain an understanding of the importance and implications of analyses using Lagrangian coordinates.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

An HPC-Based Hydrothermal Finite Element Simulator for Modeling Underground Geothermal Behavior with Example Simulations on The Treasure Island and UC Berkeley Campus

This submission contains the source code of the Hydrothermal Finite Element Simulator used for the Treasure Island and UC Berkeley campus geothermal simulation. It contains a report that summarizes the development and validation of this Hydrothermal Finite Element Simulator, with a case study on Treasure Island site. It also contains a report that investigates the feasibility of upgrading the existing campus energy delivery system at UC Berkeley to a fifth-generation district heating and cooling system that includes geothermal heat/cold storage.

15 GEOTHERMAL ENERGY↗
DOE OSTI2020

Fluctuation dynamo in a weakly collisional plasma

The turbulent amplification of cosmic magnetic fields depends upon the material properties of the host plasma. In many hot, dilute astrophysical systems, such as the intracluster medium (ICM) of galaxy clusters, the rarity of particle–particle collisions allows departures from local thermodynamic equilibrium. These departures – pressure anisotropies – exert anisotropic viscous stresses on the plasma motions that inhibit their ability to stretch magnetic-field lines. We present an extensive numerical study of the fluctuation dynamo in a weakly collisional plasma using magnetohydrodynamic (MHD) equations endowed with a field-parallel viscous (Braginskii) stress. When the stress is limited to values consistent with a pressure anisotropy regulated by firehose and mirror instabilities, the Braginskii-MHD dynamo largely resembles its MHD counterpart, particularly when the magnetic field is dynamically weak. If instead the parallel viscous stress is left unabated – a situation relevant to recent kinetic simulations of the fluctuation dynamo and, we argue, to the early stages of the dynamo in a magnetized ICM – the dynamo changes its character, amplifying the magnetic field while exhibiting many characteristics reminiscent of the saturated state of the large-Prandtl-number ( ${Pm}\gtrsim {1}$ ) MHD dynamo. We construct an analytic model for the Braginskii-MHD dynamo in this regime, which successfully matches simulated dynamo growth rates and magnetic-energy spectra. A prediction of this model, confirmed by our numerical simulations, is that a Braginskii-MHD plasma without pressure-anisotropy limiters will not support a dynamo if the ratio of perpendicular and parallel viscosities is too small. This ratio reflects the relative allowed rates of field-line stretching and mixing, the latter of which promotes resistive dissipation of the magnetic field. Finally, in all cases that do exhibit a viable dynamo, the generated magnetic field is organized into folds that persist into the saturated state and bias the chaotic flow to acquire a scale-dependent spectral anisotropy.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Electric field dependent local structure of ( K x N a 1 - x ) 0.5 B i 0.5 Ti O 3

The in situ x-ray pair-distribution function (PDF) characterization technique has been used to study the behavior of (K x Na 1–x ) 0.5 Bi 0.5 TiO 3 , as a function of electric field. As opposed to conventional x-ray Bragg diffraction techniques, PDF is sensitive to local atomic displacements, detecting local structural changes at the angstrom to nanometer scale. Several field-dependent ordering mechanisms can be observed in x = 0.15, 0.18 and at the morphotropic phase boundary composition x = 0.20. X-ray total scattering shows suppression of diffuse scattering with increasing electric-field amplitude, indicative of an increase in structural ordering. Analysis of PDF peaks in the 3–4-Å range shows ordering of Bi-Ti distances parallel to the applied electric field, illustrated by peak amplitude redistribution parallel and perpendicular to the electric-field vector. A transition from < 110 > to < 112 >-type off-center displacements of Bi relative to the neighboring Ti atoms is observable with increasing x. Analysis of PDF peak shift with electric field shows the effects of Bi-Ti redistribution and onset of piezoelectric lattice strain. Furthermore, the combination of these field-induced ordering mechanisms is consistent with local redistribution of Bi-Ti distances associated with domain reorientation and an overall increase in order of atomic displacements.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2019

Improved renormalization group computation of likelihood functions for cosmological data sets

Evaluation of likelihood functions for cosmological large scale structure data sets (including CMB, galaxy redshift surveys, etc.) naturally involves marginalization, i.e., integration, over an unknown underlying random signal field. Recently, I showed how a renormalization group method can be used to carry out this integration efficiently by first integrating out the smallest scale structure, i.e., localized structure on the scale of differences between nearby data cells, then combining adjacent cells in a coarse graining step, then repeating this process over and over until all scales have been integrated. Here I extend the formulation in several ways in order to reduce the prefactor on the method's linear scaling with data set size. The key improvement is showing how to integrate out the difference between specific adjacent cells before summing them in the coarse graining step, compared to the original formulation in which small-scale fluctuations were integrated more generally. I suggest some other improvements in details of the scheme, including showing how to perform the integration around a maximum likelihood estimate for the underlying random field. In the end, an accurate likelihood computation for a million-cell Gaussian test data set runs in two minutes on my laptop, with room for further optimization and straightforward parallelization.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Solving Seismic Wave Equations on Variable Velocity Models With Fourier Neural Operator

Here, in the study of subsurface seismic imaging, solving the acoustic wave equation is a pivotal component in existing models. The advancement of deep learning (DL) enables solving partial differential equations (PDEs), including wave equations, by applying neural networks to identify the mapping between the inputs and the solution. This approach can be faster than traditional numerical methods when numerous instances are to be solved. Previous works that concentrate on solving the wave equation by neural networks consider either a single velocity model or multiple simple velocity models, which is restricted in practice. Instead, inspired by the idea of operator learning, this work leverages the Fourier neural operator (FNO) to effectively learn the frequency domain seismic wavefields under the context of variable velocity models. We also propose a new framework paralleled FNO (PFNO) for efficiently training the FNO-based solver given multiple source locations and frequencies. Numerical experiments demonstrate the high accuracy of both FNO and PFNO with complicated velocity models in the OpenFWI datasets. Furthermore, the cross-dataset generalization test verifies that PFNO adapts to out-of-distribution velocity models. Finally, PFNO admits higher computational efficiency on large-scale testing datasets than the traditional finite-difference method. The aforementioned advantages endow the FNO-based solver with the potential to build powerful models for research on seismic waves.

58 GEOSCIENCES↗
DOE OSTI2026

Scalable Computation of Topological Abstractions for Scalar Data

Topological data analysis has become an important tool for large scale scalar data analysis and visualization, efficiently extracting the inherent structure and features of interest of the data. However, with growing dataset sizes and complexity, it is increasingly becoming infeasible to compute topological abstractions of interest in serial and on single machines. This paper presents the state of the art in the scalable computation of topological abstractions on scalar data, in shared memory parallel on single machines, and in distributed memory parallel on multiple machines. We highlight results for set‐based, graph‐based and complex‐based abstractions and organize the state of the art based on this taxonomy. The paper identifies parallelization and distribution techniques common in topological algorithms and highlights further areas of interest with underdeveloped efforts.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Microchannel-based Membrane-less Extraction of Li from Unconventional Lithium Sources & the Separation of REE

This final report provides an overview of the Project's entire duration, covering July 1, 2021 to December 31, 2023. It primarily focuses on the achievements, technological developments, and unique challenges the team faced while working on separating and extracting Lithium from produced waters. The project's primary aim was to create an integrated, high-throughput, membrane-less, and modular microfluidic platform that could extract Lithium from unconventional sources. We have successfully met all goals and milestones envisioned in the SOPO document. The most critical primary milestones, including the Go-No-Go milestone (refer to the Gantt chart in the Appendices), were successfully accomplished. We demonstrated phase separation (>90%) and extraction (>85%) performance in the MPSE using synthetic, and representative produced water composition feed at 50 ml/min total flow through MPSE 36. We have also performed a parametric study of the MPSE operations, beyond the scope of SOPO, exploring operating conditions of current and broader interest. The extended investigation of operational parameters is concurrent with our efforts to seek further development of the MPSE technology beyond the scope of the Project. Along these lines of development, we have made efforts to be responsive to DOE calls for technological developments of other types of resources (beyond PW) for the recovery of Critical Materials and higher TRL development (beyond TRL 4). During the work on this Project, we developed and implemented three innovative technical approaches that emerged from our efforts to successfully meet the Project milestones. The innovative & original technical approaches developed and implemented in this Project are now the contributions to process engineering that could be clearly credited to the Project. First, Convergent Design Approach is a comprehensive feedforward & feedback loop of four design phases: i) design for functionality, ii) design for manufacturing, iii) design for sustainability, and iv) design for market. Next was Process Intensification. A major aim of this Project was to create an innovative phase separation & extraction microscale-based technology for Li separation – thus the words microchannel-based in the Project title. A microscale-based technology is intrinsically in the center of the Process Intensification domain as defined by its unique principles. Therefore, Process Intensification was implicitly envisioned in the Project’s SOPO. Lastly, Time Scale Analysis is a novel tool for discovering the needs and directions of Process Intensification implementations in any process technology. This Project is fully credited for developing and implementing the three novel technical approaches mentioned above. These are general contributions to process engineering that emerged from this Project. Beyond the original SOPO scope, the OSU-U.Pitt research group utilized a Convergent Design methodology, integrating first-principles mathematical modeling with experimental validation on the Minimum Development Vehicle. By creating these Digital Twins, the team rapidly assessed manufacturing iterations to support TEA analysis. This framework further enabled the development of advanced Surface Modification Techniques, where hydrophobic and oleophobic coating strategies were optimized via Digital Twin tools and validated through rigorous 100-hour longevity testing. TEA Analysis: The closing efforts of this Project were focused on the TEA analysis. TEA analysis had two primary functions: i) enabling critical assessments of design variations withing 10 the Concurrent Design Approach, thus enabling evolution of the MPSE design to reach faster- better-cheaper alternatives; and ii) to create a bridge between the accomplishments of this Project and future projects of higher TRL, beyond TRL 6 level. It is important to note that the TEA model created in the Project stirred the technological solutions for the recovery of critical materials toward a vision of a very profitable modular plant that has unique zero-waste water discharge signature. More importantly, thanks to our experimental performance data and conservative assumptions, the TEA model predicts minimal technological and investment risks. Low cost of a modular unit of a nominal capacity of [1000 tons of Li 2 CO 3 /year] positions the MPSE based technology within the reach of community investors, thus offering a paradigm shift in the development of critical technologies. The project successfully navigated two primary challenges: solvent selection and manufacturing adaptation. Restricted by the SOPO to existing literature for lithium recovery, the team identified a critical need for a "material excellence program" to develop next-generation solvents, eventually concluding with a preliminary investigation into promising Ionic Liquids (ILs). Simultaneously, COVID-19 supply chain disruptions forced a pivot from traditional manufacturing to advanced additive methods at ATAMI-OSU. By transitioning from stainless steel to 3D-printed polymer substrates, the team achieved a transformative three-order-of- magnitude reduction in manufacturing costs and compressed prototyping timelines from several months to just two days. The MPSE technology offers significant energy, environmental, and economic advantages by overcoming the traditional bottlenecks of phase-separation hardware and contactor size. Unlike conventional mixer-settlers or membrane-based systems, MPSE operates without moving parts or fouling-prone membranes, achieving robust performance even with challenging, viscous, or particulate-heavy feeds. Key performance metrics include an energy intensity reduction of 5–50x (3–40 kJ/m 3 ) compared to incumbent technologies and a dramatic reduction of processing time to under 60 seconds, which drastically reduces the physical plant footprint. These technical efficiencies translate into superior economic outcomes; for a 100 t/year Li 2 CO 3 facility, implementing MPSE is projected to nearly halve contactor CAPEX (from $\$$6.08M to $\$$3.01M) and significantly increase the project's Net Present Value (NPV), derisking new investment and enabling distributed critical-mineral processing configurations. The commercialization of MPSE technology is being spearheaded by Vigsur Dynamics Inc., which has adopted a structured, parallel approach to technical and business development since its formation in January 2026. Following extensive customer discovery and engagement with the Oregon State University accelerator, Vigsur Dynamics is working to establish a business model that transitions from pilot demonstrations to modular hardware sales, ultimately aiming for a "build-own-operate" service strategy. Current technical milestones—including 100 hours of continuous operation, superior energy efficiency, and successful 6-unit modular scale-up— provide a foundation for this transition. Backed by ongoing IP licensing and a growing network of industrial and venture advisors, the company is actively de-risking the platform to replace conventional mixer-settler systems in the critical minerals market.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Automated pipeline framework for processing of large-scale building energy time series data

Commercial buildings account for one third of the total electricity consumption in the United States and a significant amount of this energy is wasted. Therefore, there is a need for “virtual” energy audits, to identify energy inefficiencies and their associated savings opportunities using methods that can be non-intrusive and automated for application to large populations of buildings. Here we demonstrate virtual energy audits applied to large populations of buildings’ time-series smart-meter data using a systematic approach and a fully automated Building Energy Analytics (BEA) Pipeline that unifies, cleans, stores and analyzes building energy datasets in a non-relational data warehouse for efficient insights and results. This BEA pipeline is based on a custom compute job scheduler for a high performance computing cluster to enable parallel processing of Slurm jobs. Within the analytics pipeline, we introduced a data qualification tool that enhances data quality by fixing common errors, while also detecting abnormalities in a building’s daily operation using hierarchical clustering. We analyze the HVAC scheduling of a population of 816 buildings, using this analytics pipeline, as part of a cross-sectional study. With our approach, this sample of 816 buildings is improved in data quality and is efficiently analyzed in 34 minutes, which is 85 times faster than the time taken by a sequential processing. The analytical results for the HVAC operational hours of these buildings show that among 10 building use types, food sales buildings with 17.75 hours of daily HVAC cooling operation are decent targets for HVAC savings. Overall, this analytics pipeline enables the identification of statistically significant results from population based studies of large numbers of building energy time-series datasets with robust results. These types of BEA studies can explore numerous factors impacting building energy efficiency and virtual building energy audits. This approach enables a new generation of data-driven buildings energy analysis at scale.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Two-Fluid and Discrete Element Modeling of a Parallel Plate Fluidized Bed Heat Exchanger for Concentrating Solar Power

A novel high-temperature particle solar receiver is developed using a light trapping planar cavity configuration. As particles fall through the cavity, the concentrated solar radiation warms the boundaries of the receiver and in turn heats the particles. Particles flow through the system, forming a fluidized bed at the lower section, leaving the system from the bottom at a constant flowrate. Air is introduced to the system as the fluidizing medium to improve particle heat transfer and mixing. A laboratory scale cavity receiver is built by collaborators at the Colorado School of Mines and their data are used for model validation. In this experimental setup, near IR quartz lamp is used to provide flux to the vertical wall of the heat exchanger. The system is modeled using the discrete element method and a continuum two-fluid method. The computational model matches the experimental system size and the particle size distribution is assumed monodisperse. A new continuum conduction model that accounts for the effects of solid concentration is implemented, and the heat flux boundary condition matches the experimental setup. Radiative heat transfer is estimated using a widely used correlation during the post-processing step to determine an overall heat transfer coefficient. The model is validated against testing data and achieves less than 30% discrepancy and a heat transfer coefficient greater than 1000 W/m2 K.

concentrating solar power↗
DOE OSTI2023

Multiphase Modeling in a Parallel Plate Fluidized Bed Receiver for Concentrating Solar Power

A novel high temperature particle solar receiver is developed by using a light trapping planar cavity configuration. As particles fall through the cavity, the concentrated solar radiation warms the boundaries of the receiver and in turn heats the particles. Particles flow through the system, forming a packed bed at the lower end, leaving the system from the bottom at a constant flow rate. Air is introduced to the system as the fluidizing medium to improve particle heat transfer and mixing. A laboratory scale cavity receiver is built and a near IR quartz lamp is used to provide flux to the vertical wall of the heat exchanger. The system is modeled using a continuum two-fluid method. The computational model matches the experimental system size and the particle size distribution is assumed monodisperse. A conduction model that accounts for the effects of solid concentration is implemented, and the heat flux boundary condition matches the experimental setup. Radiative heat transfer is estimated using a widely used correlation during the post-processing step to determine an overall heat transfer coefficient. The model is validated against testing data and achieves less than 30% discrepancy and a heat transfer coefficient greater than 1000 W/m2K.

CSP↗
DOE OSTI2023

Simulating the Impact of Dynamic Rerouting on Metropolitan-scale Traffic Systems

The rapid introduction of mobile navigation aides that use real-time road network information to suggest alternate routes to drivers is making it more difficult for researchers and government transportation agencies to understand and predict the dynamics of congested transportation systems. Computer simulation is a key capability for these organizations to analyze hypothetical scenarios; however, the complexity of transportation systems makes it challenging for them to simulate very large geographical regions, such as multi-city metropolitan areas. In this article, we describe enhancements to the Mobiliti parallel traffic simulator to model dynamic rerouting behavior with the addition of vehicle controller actors and vehicle-to-controller reroute requests. The simulator is designed to support distributed-memory parallel execution using discrete event simulation and be scalable on high-performance computing platforms. We demonstrate the potential of the simulator by analyzing the impact of varying the population penetration rate of dynamic rerouting on the San Francisco Bay Area road network. Using high-performance parallel computing, we can simulate a day in the San Francisco Bay Area with 19 million vehicle trips with 50 percent dynamic rerouting penetration over a road network with 0.5 million nodes and 1 million links in less than three minutes. We present a sensitivity study on the dynamic rerouting parameters, discuss the simulator’s parallel scalability, and analyze system-level impacts of changing the dynamic rerouting penetration. Furthermore, we examine the varying effects on different functional classes and geographical regions and present a validation of the simulation results compared to real-world data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Boundary-directed epitaxy of block copolymers

Directed self-assembly of block copolymers (BCPs) enables nanofabrication at sub-10 nm dimensions, beyond the resolution of conventional lithography. However, directing the position, orientation, and long-range lateral order of BCP domains to produce technologicallyuseful patterns is a challenge. Here, we present a promising approach to direct assembly using spatial boundaries between planar, low-resolution regions on a surface with different composition. Pairs of boundaries are formed at the edges of isolated stripes on a background substrate. Vertical lamellae nucleate at and are pinned by chemical contrast at each stripe/ substrate boundary, align parallel to boundaries, selectively propagate from boundaries into stripe interiors (whereas horizontal lamellae form on the background), and register to wide stripes to multiply the feature density. Ordered BCP line arrays with half-pitch of 6.4 nm are demonstrated on stripes >80 nm wide. Boundary-directed epitaxy provides an attractive path towards assembling, creating, and lithographically defining materials on sub-10 nm scales.

36 MATERIALS SCIENCE↗
DOE OSTI2019

Effect of strain on charge density wave order in the Holstein model

In this work, we investigate charge ordering in the Holstein model in the presence of anisotropic hopping, t x ,t y =1-δ,1+δ, as a model of the effect of strain on charge-density-wave (CDW) materials. Using quantum Monte Carlo simulations, we show that the CDW transition temperature is relatively insensitive to moderate anisotropy δ≲0.3, but begins to decrease more rapidly at δ≳0.4. However, the density correlations, as well as the kinetic energies parallel and perpendicular to the compressional axis, change significantly for moderate δ. Accompanying mean-field theory calculations show a similar qualitative structure, with the transition temperature relatively constant at small δ, and a more rapid decrease for larger strains. We also obtain the density of states N(ω), which provides clear signal of the charge ordering transition at large strain, where finite size scaling of the charge structure factor is extremely difficult because of the small value of the order parameter.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2023

Progress and R/D challenges for FCC-ee SRF

The FCC-ee machines present a huge challenge for the RF systems, which need to be adapted to very diverse beam conditions going from moderate energy and high current for the Z machine to high energy and low beam current for the ttbar. This inverse scaling results naturally from a fixed budget for the synchrotron radiation, which the SRF cavities need to compensate. A global solution was elaborated for the FCC Conceptual Design Report (Abada in Eur Phys J Spec Top 228):261–623, 2019), and is referred here as the baseline. Recently, further studies have led to a new optimized baseline, still based on traditional elliptical cavities. In parallel, a novel concept, named the Slotted Waveguide ELLiptical (SWELL), was proposed with the potential of greatly simplified logistics and reduced costs. Under several aspects, all these changes call for enhanced performance of the RF systems. A vigorous R&D program has therefore continued since the publication of the CDR, with the aim of pushing the performance and demonstrating the feasibility of a more advanced baseline and, more recently, of the SWELL option. The progress and challenges of this ambitious program were presented in the dedicated SRF sessions at FCC week 2022 and are summarized in this paper.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Initial Developments in Modeling Graphite Behavior

Graphite is used in High Temperature Reactor designs as a moderator, reflector and core material responsible for protecting the fuel and maintaining structural stability. From a safety perspective, it is imperative to be able to predict how graphite will behave under reactor operating conditions which may compromise the core’s structural integrity or fuel’s safety performance. The following report summarizes recent modeling work undertaken at Idaho National Laboratory which focuses on graphite degradation behavior. One phenomenon which is explored in this report and can cause significant degradation to graphite properties is oxidation. Oxidation behavior in graphite is highly temperature dependent. At low temperatures, oxygen can fully penetrate a component and cause a homogeneous damage profile. At high temperatures, only the graphite near the surface of a component is affected. Understanding and modeling this temperature dependence is essential to predicting graphite behavior. In this report, three graphite models are discussed. The first model is used to investigate strength loss after low temperature oxidation. It does this by generating an approximate graphite microstructure then determining the required load to cause crack propagation. It has been shown that the model reproduces the quasi brittle stress versus strain relationship observed in graphite. Strength loss results from the model are shown to match experimental values. The second model investigates strength loss under high temperature oxidation conditions. This model is more applicable to full scale graphite components where the oxidation damage is often inhomogeneous throughout a component. The third model computes stresses in graphite under reactor conditions. Specifically, the model incorporates the effect of creep, irradiation dose, thermal strains, and oxidation in order to predict internal stresses in a component. All of these models are implemented in the Multiphysics Object-Oriented Simulation Environment (MOOSE), an open source, parallel finite element framework.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading

Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first exascale computer, Frontier, comprising AMD MI250x GPUs with HPE's Slingshot interconnect and operated by the US Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCm library calls. With 99% of computing power on Frontier being on the GPU, leaving the CPU idle leads to a net performance gain via avoiding the overhead of data movement between host and device except when needed for some I/O purposes. Memory footprint including the size of communication buffers for MPI_ALLTOALL is managed carefully to maximize the largest problem size possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from 2048 3 on a single node to 32768 3 on 4096 or 8192 nodes out of 9408 on the system. Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are implemented. The present code suite (labeled by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance attained is highly favorable in both weak scaling and strong scaling, with notable departures only for 2048 3 where communication is entirely intra-node, and for 32768 3 , where a challenge due to small message sizes does arise. Communication performance is addressed further using a lightweight test code that performs all-to-all communication in a manner matching the full turbulence simulation code. Performance at large problem sizes is affected by both small message size due to high node counts as well as dragonfly network topology features on the machine, but is consistent with official expectations of sustained performance on Frontier. Overall, although not perfect, the scalability achieved at the extreme problem size of 32768 3 (and up to 8192 nodes — which corresponds to hardware rated at just under 1 exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled by this code and its extensions are to be reported separately in the near future.

3D fast Fourier transform↗