Search NASA⌕ Search

Engineering topics

Harris, Austin

Publications and source records attributed to Harris, Austin.

An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers

Dragonfly-based networks are an extensively deployed network topology in large-scale high-performance computing due to their cost-effectiveness and efficiency. The US will soon have three Exascale supercomputers for leadership class workloads deployed using dragonfly networks. Compared to indirect networks of similar scale, the dragonfly network has considerably reduced cable lengths, cable counts, and switch counts, resulting in significant network cost savings for a given system size, however, these cost reductions result in reduced global minimal paths and more challenging routing. Additionally, large scale dragonfly networks often require a taper at the global link level, resulting in less bisection bandwidth than is achievable in other traditional non-blocking topologies of equivalent scale. While dragonfly networks have been extensively studied, they have yet to be fully evaluated in an extreme scale (i.e., exascale) system that targets capability workloads. In this paper, we present the results of the first large scale evaluation of a dragonfly network on an exascale system (Frontier) and compare its behavior to a similar scale fat-tree network on a previous generation TOP500 system (Summit). This evaluation aims to determine the effect of network cost optimizations by measuring a tapered topology’s impact on capability workloads. Our evaluation is based on a collection of synthetic microbenchmarks, mini-apps, and full scale applications. It compares the scaling efficiencies of each benchmark between the dragonfly-based Frontier and the fat-tree-based Summit systems. Our results show that a dragonfly network is $\sim \mathbf{3 0 \%}$ more cost efficient than a fat-tree topology, which amortizes to $\sim 3 \%$ of an exascale system cost. Furthermore, while tapered dragonfly networks impose significant tradeoffs, the impacts are not as broad as initially thought and are mostly seen in applications with global communication patterns, particularly all-to-all (e.g., FFT-based algorithms), but also local communication patterns (e.g., nearest-neighbor algorithms) that are sensitive to network performance variability.

Khan, Awais↗

Flash-X

Flash-X is a highly composable multiphysics software system that can be used to simulate physical phenomena in several scientific domains. It is derived from FLASH, which has been a community code for several communities over the last 20 years. The Flash-X architecture has been redesigned to be compatible with increasingly heterogeneous hardware platforms. A part of the redesign is a utilizing a newly designed performance portability layer that is language agnostic.

Dubey, Anshu [Argonne National Laboratory (ANL), A↗

Application Experiences on a GPU-Accelerated Arm-based HPC Testbed

This paper assesses and reports the experience of ten teams working to port, validate, and benchmark several High Performance Computing applications on a novel GPU-accelerated Arm testbed system. The testbed consists of eight NVIDIA Arm HPC Developer Kit systems, each one equipped with a server-class Arm CPU from Ampere Computing and two data center GPUs from NVIDIA Corp. The systems are connected together using InfiniBand interconnect. The selected applications and mini-apps are written using several programming languages and use multiple accelerator-based programming models for GPUs such as CUDA, OpenACC, and OpenMP offloading. Working on application porting requires a robust and easy-to-access programming environment, including a variety of compilers and optimized scientific libraries. The goal of this work is to evaluate platform readiness and assess the effort required from developers to deploy well-established scientific workloads on current and future generation Arm-based GPU-accelerated HPC systems. The reported case studies demonstrate that the current level of maturity and diversity of software and tools is already adequate for large-scale production deployments.

Elwasif, Wael↗

Accelerating Flash-X Simulations with Asynchronous I/O

Most high-fidelity physics simulation codes, such as Flash-X, need to save intermediate results (checkpoint files) to restart or gain insights into the evolution of the simulation. These simulation codes save such intermediate files synchronously, where computation is stalled while the data is written to storage. Depending on the problem size and computational requirements, this file write time can be a substantial portion of the total simulation time. In order to hide the I/O latency of checkpointing, asynchronous I/O methods have been introduced. These methods use background threads for performing I/O while the main threads continue with the simulation. The usage of background threads can compete for resources on the node as well as with communication. In this paper, we evaluate the overheads and the overall benefit of asynchronous I/O in HDF5 to simulations. Results from real-world high-fidelity simulations on the Summit supercomputer show that I/O operation is overlapped with application communication or computation or both, effectively hiding some or all of the I/O latency. Our evaluation shows that while using asynchronous I/O adds overhead to the application, the I/O time reduction is more significant, resulting in overall up to 1.5X performance speedup.

Jain, Rajeev↗

Performance Debugging and Tuning of Flash-X with Data Analysis Tools

State-of-the-art multiphysics simulations running on large scale leadership computing platforms have many variables contributing to their performance and scaling behavior. We recently encountered an interesting performance anomaly in Flash-X, a multiphysics multicomponent simulation software, when characterizing its performance behavior on several large-scale HPC platforms. The anomaly was tracked down to the interaction between the use of dynamic allocation of scratch data and data locality in the cache hierarchy. In this paper we present the details of unexpected performance variability of Flash-X, its extensive analysis using the performance measurement tool TAU to collect the data and Python data analysis libraries to explore the data, and our insights from this experience. In this process, we discovered and removed or mitigated two additional performance limiting bottlenecks for performance tuning.

Huck, Kevin↗

thornado-transport: Anderson- and GPU-accelerated nonlinear solvers for neutrino-matter coupling

Algorithms for neutrino-matter coupling in core-collapse supernovae (CCSNe) are investigated in the context of a spectral two-moment model, which is discretized in space with the discontinuous Galerkin method, integrated in time with implicit-explicit (IMEX) methods, and implemented in the toolkit for high-order neutrino-radiation hydrodynamics (thornado). The model considers electron neutrinos and antineutrinos and tabulated opacities from Bruenn (1985), which includes neutrino-electron scattering and pair processes. The nonlinear system arising from implicit time discretization of the equations governing neutrino-matter coupling is iterated to convergence using Anderson-accelerated fixed-point methods, which avoid formation of Jacobians and inversion of dense linear systems. Numerical experiments show that, for a given tolerance, a nested iteration scheme which aims to reduce opacity evaluations can lower the computational cost. Our initial port to GPUs, using both OpenMP and OpenACC, shows an overall speedup of up to ~ 100× when compared to results using a single CPU core. These results indicate that the algorithms implemented in thornado are well-suited to GPU acceleration.

Laiu, Paul↗

On the character of turbulent-like flows in self-consistent models of core-collapse supernovae

Neutrino-driven convection plays a crucial role in the development of core-collapse supernova (CCSN) explosions. However, the complex mechanism that triggers the shock revival and the subsequent explosion has remained inscrutable for many decades. Multidimensional simulations suggest that the growth of fluid instabilities, especially development of convection, will determine the morphology of the explosion. We have performed 3D simulations using spherical-polar coordinates covering a reduced angular extent (90° computational domain), and with angular resolutions of 2°, 1°, 1/2°, and 1/4°, to study the development of turbulent-like flows in core-collapse supernova explosions on a time scale of order 100 ms. We have employed the multi-physics Chimera code that includes detailed nuclear physics and spectral neutrino transport. Coarse resolution models do not develop an inertial range, presumably due to the bottleneck effect, such that the energy is prevented from cascading down to small scales and tends to accumulate at large scales. High-resolution models instead, start to recover the k −5/3 scaling of Kolmogorov's theory. Stochasticity and few simulation samples limit our ability to predict the development of explosions. Over the simulated time period, our models show no clear trend in improving (or diminishing) conditions for explosion as the angular resolution is increased. However, we find that disordered flow provides an effective pressure, as characterized by the Reynolds stress, behind the shock. This contribution, ~40%–50% of the thermal pressure, aids shock revival and thus the development of the explosion. Finally, we show that the kinetic energy power spectrum of reduced angular extent and full 4π models are consistent, thus indicating that a 90° computational domain is an adequate configuration to study the character of turbulence in CCSNe.

Casanova Bustamante, Jordi↗

Learning About Nucleosynthesis from Multi-dimensional Simulations of Core-Collapse Supernovae

For more than two decades, we have understood that the development of a successful core-collapse supernova is inextricably linked to neutrino heating and three dimensional fluid flows, with large scale hydrodynamic instabilities allowing successful explosions that spherical symmetry would prevent. Unfortunately, our understanding of the nucleosynthesis that occurs in these supernovae, and therefore the impact of supernovae on galactic chemical evolution, has generally ignored much that we have learned about the central engine of these supernovae over the past two decades. Now, with two and three dimensional simulations of core-collapse supernovae run to sufficient duration, we are learning how the multi-dimensional, neutrino-driven character of the explosions directly impacts the nucleosynthesis and other observables of core-collapse supernovae. Here we focus on lessons from multi-dimensional models which implement realistic nuclear reaction networks.

Hix, W Raphael↗

Multi-dimensional simulations of core-collapse supernova explosions with CHIMERA

Unraveling the core-collapse supernova (CCSN) mechanism is a problem that remains essentially unsolved despite more than four decades of effort. Spherically symmetric models with otherwise high physical fidelity generally fail to produce explosions, and it is widely accepted that CCSNe are inherently multi-dimensional. Progress in realistic modeling has occurred recently through the availability of petascale platforms and the increasing sophistication of supernova codes. We will discuss our most recent work on understanding neutrino-driven CCSN explosions employing multi-dimensional neutrino-radiation hydrodynamics simulations with the Chimera code. We discuss the inputs and resulting outputs from these simulations, the role of neutrino radiation transport, and the importance of multi-dimensional fluid flows in shaping the explosions. We also highlight the production of 48Ca in long-running Chimera simulations.

Messer II, Bronson↗

The ν process in the innermost supernova ejecta

The neutrino-induced nucleosynthesis (ν process) in supernova explosions of massive stars of solar metallicity with initial main sequence masses between 13 and 30 M⊙ has been studied with an analytic explosion model using a new extensive set of neutrino-nucleus cross-sections and spectral properties that agree with modern supernova simulations. The production factors for the nuclei 7Li, 11B, 19F, 138La and 180Ta, are still significantly enhanced but do not reproduce the full solar abundances. We study the possible contribution of the innermost supernova eject to the production of the light elements 7Li and 11B with tracer particles based on a 2D supernova simulation of a 12 M⊙ progenitor and conclude, that a contribution exists but is negligible for the total yield for this explosion model.

Sieverding, Andre↗

The Multidimensional Character of Nucleosynthesis in Core-Collapse Supernovae

Observations of supernovae and their remnants reveal highly aspherical distributions of the newly-formed elements that are the dying stars’ contributions to the interstellar medium. Modern simulations of the supernova’s neutrino-powered central engine reveal that these inhomogeneities originate in the first seconds of the explosions. Yet, much of our understanding of supernova nucleosynthesis is based on spherically symmetric models of the explosions. Recent simulations, combining high-fidelity treatments of the neutrino field that drives the explosion, the multidimensional fluid flow that taps this energy source, and the thermonuclear kinetics responsible for the composition of the ejecta, are revealing the limitations of our spherically symmetric understanding. Here, we highlight these recent results to presage the changes in our understanding of supernova nucleosynthesis that will result from a full appreciation of the multidimensional character of core-collapse supernovae.

Hix, W Raphael↗