Search NASA⌕ Search

Engineering topics

Almgren, Ann S.

Publications and source records attributed to Almgren, Ann S..

The Pele Simulation Suite for Reacting Flows at Exascale

In this work, we present the Pele suite of software tools for compressible and incompressible reacting flows. The Pele suite leverages several different libraries, notably AMReX and SUNDIALS, to achieve performance portability on heterogeneous computing architectures across the supercomputing landscape. The Pele suite is comprised of PeleC, a compressible reacting flow block-structured adaptive mesh refinement solver, PeleLMeX, a low-Mach number reacting flow block-structured adaptive mesh refinement solver, Pele-Physics, a library for transport, thermodynamics, finite rate chemistry, soot, spray and radiation physics. The objective of this paper is (i) to present the code development efforts necessary to achieve highly effective and scalable applications for exascale machines and (ii) to detail the performance results of the Combustion-Pele project applications on Oak Ridge National Laboratory's Frontier. We show good weak and strong scaling results for both PeleC and PeleLMeX up to more than 50 billion cells on more than 4096 Frontier graphics processing unit nodes. We also present a capability demonstration simulation of a dual-fuel pulse compression ignition engine (six adaptive mesh refinement levels, and 60 billion cells or 2.1 trillion degrees of freedom) on Frontier, to date one of the largest simulations performed on the first exascale-class supercomputer.

adaptive mesh refinement↗

BMX: Biological modelling and interface exchange

Abstract High performance computing has a great potential to provide a range of significant benefits for investigating biological systems. These systems often present large modelling problems with many coupled subsystems, such as when studying colonies of bacteria cells. The aim to understand cell colonies has generated substantial interest as they can have strong economic and societal impacts through their roles in in industrial bioreactors and complex community structures, called biofilms, found in clinical settings. Investigating these communities through realistic models can rapidly exceed the capabilities of current serial software. Here, we introduce BMX, a software system developed for the high performance modelling of large cell communities by utilising GPU acceleration. BMX builds upon the AMRex adaptive mesh refinement package to efficiently model cell colony formation under realistic laboratory conditions. Using simple test scenarios with varying nutrient availability, we show that BMX is capable of correctly reproducing observed behavior of bacterial colonies on realistic time scales demonstrating a potential application of high performance computing to colony modelling. The open source software is available from the zenodo repository https://doi.org/10.5281/zenodo.8084270 under the BSD-2-Clause licence.

97 MATHEMATICS AND COMPUTING↗

In situ feature analysis for large-scale multiphase flow simulations

The study of multiphase flow is essential for designing chemical reactors such as fluidized bed reactors (FBR), as a detailed understanding of hydrodynamics is critical for optimizing reactor performance and stability. An FBR allows scientists to conduct different types of chemical reactions involving multiphase materials, especially interaction between gas and solids. During such complex chemical processes, the formation of void regions in the reactor, generally termed as bubbles, is an important phenomenon. The study of these bubbles has a deep implication in predicting the reactor’s overall efficiency. But physical experiments needed to understand bubble dynamics are costly and non-trivial due to the technical difficulties involved and harsh working conditions of the reactors. Therefore, to study such chemical processes and bubble dynamics, a state-of-the-art computational simulation MFIX-Exa is being developed. Despite the proven accuracy of MFIX-Exa in modeling bubbling phenomena, the large-scale output data prohibits the use of traditional post hoc analysis capabilities in both storage and I/O time. Herein, to address these issues and allow the application scientists to explore the bubble dynamics in an efficient and timely manner, we have developed an end-to-end analytics pipeline that enables in situ detection of bubbles, followed by a flexible post hoc visual exploration methodology of bubble dynamics. The proposed method enables interactive analysis of bubbles, along with quantification of several bubble characteristics, enabling experts to understand the bubble interactions in detail. Positive feedback from the experts has indicated the efficacy of the proposed approach for exploring bubble dynamics in very-large-scale multiphase flow simulations.

97 MATHEMATICS AND COMPUTING↗

Hurricane‐Like Vortices in Conditionally Unstable Moist Convection

Abstract This study investigates the emergence of hurricane‐like vortices in idealized simulations of rotating moist convection. A Boussinesq atmosphere with simplified thermodynamics for phase transitions is forced by prescribing the temperature and humidity at the upper and lower boundaries. The governing equations are solved numerically using a variable‐density incompressible Navier‐Stokes solver with adaptive mesh refinement to explore the behavior of moist convection under a broad range of conditions. In the absence of rotation, convection aggregates into active patches separated by large unsaturated regions. Rotation modulates this statistical equilibrium state so that the self‐aggregated convection organizes hurricane‐like vortices. The warm and saturated air converges to the center of the vortices, and the latent heat released through the upwelling, forms the warm core structure. These hurricane‐like vortices share characteristics similar to tropical cyclones in the earth's atmosphere. The hurricane‐like vortices occur under conditionally unstable conditions where the potential energy given at the boundaries is large enough, corresponding to a moderate rate of rotation. This regime shares many similar characteristics to the tropical atmosphere indicating that the formation of intense meso‐scale vortices is a general characteristic of rotating moist convection. The model used here does not include any interactions with radiation, wind‐evaporation feedback, or cloud microphysics, indicating that, while these processes may be relevant for tropical cyclogenesis in the Earth atmosphere, they are not its primary cause. Instead, our results confirm that the formation and maintenance of hurricane‐like vortices involve a combination of atmospheric dynamics under the presence of rotation and of phase transitions.

54 ENVIRONMENTAL SCIENCES↗

A weighted state redistribution algorithm for embedded boundary grids

State redistribution is an algorithm that stabilizes cut cells for embedded boundary grid methods. This work extends the earlier algorithm in several important ways. First, state redistribution is extended to three spatial dimensions. Second, we discuss several algorithmic changes and improvements motivated by the more complicated cut cell geometries that can occur in higher dimensions. In particular, we introduce a weighted version with less dissipation in an easily generalizable framework. Third, we demonstrate that state redistribution can also stabilize a solution update that includes both advective and diffusive contributions. Notably, the stabilization algorithm is shown to be effective for incompressible as well as compressible reacting flows. Finally, we discuss the implementation of the algorithm for several exascale-ready simulation codes based on AMReX, demonstrating ease of use in combination with domain decomposition, hybrid parallelism and complex physics.

97 MATHEMATICS AND COMPUTING↗

A coupled discontinuous Galerkin-Finite Volume framework for solving gas dynamics over embedded geometries

Herein, we present a computational framework for solving the equations of inviscid gas dynamics using structured grids with embedded geometries. The novelty of the proposed approach is the use of high-order discontinuous Galerkin (dG) schemes and a shock-capturing Finite Volume (FV) scheme coupled via an hp adaptive mesh refinement (hp-AMR) strategy that offers high-order accurate resolution of the embedded geometries. The hp-AMR strategy is based on a multi-level block-structured domain partition in which each level is represented by block-structured Cartesian grids and the embedded geometry is represented implicitly by a level set function. The intersection of the embedded geometry with the grids produces the implicitly-defined mesh that consists of a collection of regular rectangular cells plus a relatively small number of irregular curved elements in the vicinity of the embedded boundaries. High-order quadrature rules for implicitly-defined domains enable high-order accuracy resolution of the curved elements with a cell-merging strategy to address the small-cell problem. The hp-AMR algorithm treats the system with a second-order finite volume scheme at the finest level to dynamically track the evolution of solution discontinuities while using dG schemes at coarser levels to provide high-order accuracy in smooth regions of the flow. On the dG levels, the methodology supports different orders of basis functions on different levels. The space-discretized governing equations are then advanced explicitly in time using high-order Runge-Kutta algorithms. Numerical tests are presented for two-dimensional and three-dimensional problems involving an ideal gas. The results are compared with both analytical solutions and experimental observations and demonstrate that the framework provides high-order accuracy for smooth flows and accurately captures solution discontinuities.

97 MATHEMATICS AND COMPUTING↗

Feature Analysis, Tracking, and Data Reduction: An Application to Multiphase Reactor Simulation MFiX-Exa for In-Situ Use Case

As we enter the exascale computing regime, powerful supercomputers continue to produce much higher amounts of data than what can be stored for offline data processing. To utilize such high compute capabilities on these machines, much of the data processing needs to happen in situ, when the full high-resolution data is available at the supercomputer memory. In this article, we discuss our MFiX-Exa simulation, which models multiphase flow by tracking a very large number of particles through the simulation domain. In one of the use cases, the carbon particles interact with air to produce carbon dioxide bubbles from the reactor. These bubbles are of primary interest to the domain experts for these simulations. For this particle-based simulation, we propose a streaming technique that can be deployed in situ to efficiently identify the bubbles, track them over time, and use them to down-sample the data with minimal loss in these features.

97 MATHEMATICS AND COMPUTING↗

The divergence of nearby trajectories in soft-sphere DEM

The n-body instability is investigated with the soft-sphere discrete element method. The divergence of nearby trajectories is quantified by the dynamical memory time. Using the inverse proportionality between the dynamical memory time and the largest Lyapunov exponent, the soft-sphere discrete element method results are compared to previous hard-sphere molecular dynamics data for the first time. Good agreement is observed at low concentrations and the degree of instability is shown to increase asymptotically with increasing spring stiffness. At particle concentrations above 30%, the soft-sphere Lyapunov exponents increase faster than the corresponding hard-sphere data. Finally, this paper concludes with a demonstration of how this case study may be used in conjunction with regression testing and code verification activities.

42 ENGINEERING↗

MFIX-Exa: A path toward exascale CFD-DEM simulations

MFIX-Exa is a computational fluid dynamics–discrete element model (CFD-DEM) code designed to run efficiently on current and next-generation supercomputing architectures. MFIX-Exa combines the CFD-DEM expertise embodied in the MFIX code—which was developed at NETL and is used widely in academia and industry—with the modern software framework, AMReX, developed at LBNL. The fundamental physics models follow those of the original MFIX, but the combination of new algorithmic approaches and a new software infrastructure will enable MFIX-Exa to leverage future exascale machines to optimize the modeling and design of multiphase chemical reactors.

97 MATHEMATICS AND COMPUTING↗

MAESTROeX: A Massively Parallel Low Mach Number Astrophysical Solver

@e present MAESTROeX, a massively parallel solver for low Mach number astrophysical flows. The underlying low Mach number equation set allows for efficient, long-time integration for highly subsonic flows compared to compressible approaches. MAESTROeX is suitable for modeling full spherical stars as well as well as planar simulations of dynamics within localized regions of a star, and can robustly handle several orders of magnitude of density and pressure stratification. Previously, we have described the development of the predecessor of MAESTROeX, called MAESTRO, in a series of papers. Here, we present a new, greatly simplified temporal integration scheme that retains the same order of accuracy as our previous approaches. We also explore the use of alternative spatial mapping of the one-dimensional base state onto the full Cartesian grid. The code leverages the new AMReX software framework for block-structured adaptive mesh refinement (AMR) applications, allowing for scalability to large fractions of leadership-class machines. Finally, using our previous studies on the convective phase of single-degenerate progenitor models of SNe Ia as a guide, we characterize the performance of the code and validate the new algorithmic features. Like MAESTRO, MAESTROeX is fully open source.

79 ASTRONOMY AND ASTROPHYSICS↗

Topology-Aware Performance Optimization and Modeling of Adaptive Mesh Refinement Codes for Exascale

Here, we introduce a topology-aware performance optimization and modeling workflow for AMR simulation that includes two new modeling tools, ProgrAMR and Mota Mapper, which interface with the BoxLib AMR framework and the SSTmacro network simulator. ProgrAMR allows us to generate and model the execution of task dependency graphs from high-level specifications of AMR-based applications, which we demonstrate by analyzing two example AMR-based multigrid solvers with varying degrees of asynchrony. Mota Mapper generates multiobjective, network topology-aware box mappings, which we apply to optimize the data layout for the example multigrid solvers. While the sensitivity of these solvers to layout and execution strategy appears to be modest for balanced scenarios, the impact of better mapping algorithms can be significant when performance is highly constrained by network hop latency. Furthermore, we show that network latency in the multigrid bottom solve is the main contributing factor preventing good scaling on exascale-class machines.

97 MATHEMATICS AND COMPUTING↗

WHITE DWARF MERGERS ON ADAPTIVE MESHES. I. METHODOLOGY AND CODE VERIFICATION

The Type Ia supernova (SN Ia) progenitor problem is one of the most perplexing and exciting problems in astrophysics, requiring detailed numerical modeling to complement observations of these explosions. One possible progenitor that has merited recent theoretical attention is the white dwarf (WD) merger scenario, which has the potential to naturally explain many of the observed characteristics of SNe Ia. To date there have been relatively few self-consistent simulations of merging WD systems using mesh-based hydrodynamics. This is the first paper in a series describing simulations of these systems using a hydrodynamics code with adaptive mesh refinement. In this paper we describe our numerical methodology and discuss our implementation in the compressible hydrodynamics code CASTRO, which solves the Euler equations, and the Poisson equation for self-gravity, and couples the gravitational and rotation forces to the hydrodynamics. Standard techniques for coupling gravitation and rotation forces to the hydrodynamics do not adequately conserve the total energy of the system for our problem, but recent advances in the literature allow progress and we discuss our implementation here. We present a set of test problems demonstrating the extent to which our software sufficiently models a system where large amounts of mass are advected on the computational domain over long timescales. Future papers in this series will describe our treatment of the initial conditions of these systems and will examine the early phases of the merger to determine its viability for triggering a thermonuclear detonation.

79 ASTRONOMY AND ASTROPHYSICS↗

A Low Mach Number Model for Moist Atmospheric Flows

A low Mach number model for moist atmospheric flows is introduced that accurately incorporates reversible moist processes in flows whose features of interest occur on advective rather than acoustic time scales. Total water is used as a prognostic variable, so that water vapor and liquid water are diagnostically recovered as needed from an exact Clausius–Clapeyron formula for moist thermodynamics. Low Mach number models can be computationally more efficient than a fully compressible model, but the low Mach number formulation introduces additional mathematical and computational complexity because of the divergence constraint imposed on the velocity field. Here in this paper, latent heat release is accounted for in the source term of the constraint by estimating the rate of phase change based on the time variation of saturated water vapor subject to the thermodynamic equilibrium constraint. Finally, the authors numerically assess the validity of the low Mach number approximation for moist atmospheric flows by contrasting the low Mach number solution to reference solutions computed with a fully compressible formulation for a variety of test problems.

54 ENVIRONMENTAL SCIENCES↗

Nyx: A MASSIVELY PARALLEL AMR CODE FOR COMPUTATIONAL COSMOLOGY

We present a new N-body and gas dynamics code, called Nyx, for large-scale cosmological simulations. Nyx follows the temporal evolution of a system of discrete dark matter particles gravitationally coupled to an inviscid ideal fluid in an expanding universe. The gas is advanced in an Eulerian framework with block-structured adaptive mesh refinement; a particle-mesh scheme using the same grid hierarchy is used to solve for self-gravity and advance the particles. Computational results demonstrating the validation of Nyx on standard cosmological test problems, and the scaling behavior of Nyx to 50,000 cores, are presented.

79 ASTRONOMY AND ASTROPHYSICS↗

Computer Code Gives Astrophysicists First Full Simulation of Star's Final Hours

The precise conditions inside a white dwarf star in the hours leading up to its explosive end as a Type Ia supernova are one of the mysteries confronting astrophysicists studying these massive stellar explosions. But now, a team of researchers, composed of three applied mathematicians at the U.S. Department of Energy's (DOE) Lawrence Berkeley National Laboratory and two astrophysicists, has created the first full-star simulation of the hours preceding the largest thermonuclear explosions in the universe. http://www.lbl.gov/cs/Archive/news091509.html

79 ASTRONOMY AND ASTROPHYSICS↗

Computer Code Gives Astrophysicists First Full Simulation of Star's Final Hours

The precise conditions inside a white dwarf star in the hours leading up to its explosive end as a Type Ia supernova are one of the mysteries confronting astrophysicists studying these massive stellar explosions. But now, a team of researchers, composed of three applied mathematicians at the U.S. Department of Energy's (DOE) Lawrence Berkeley National Laboratory and two astrophysicists, has created the first full-star simulation of the hours preceding the largest thermonuclear explosions in the universe.

79 ASTRONOMY AND ASTROPHYSICS↗