Search NASA⌕ Search

SEARCH · Search NASA

Results for “massively parallel simulations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Coupling hydrodynamics to a rigid-body motion solver for fluid-structure interaction [Slides]

Quinoa is a massively parallel computational fluid dynamics with multi-material and programmed burn capabilities developed from Programmatic and LDRD funds. Overset is a well-established method of using an “inset” mesh that communicates with a background mesh. Solutions transfer freely from one to another and operate as boundary conditions on the opposite mesh. Either mesh can move at specified velocity! This project couples the existing Mesh-to-mesh transfer with Quinoa and demonstrates use of resulting Overset solver in computational simulation of blast effects on a re-entry body for survivability assessments.

42 ENGINEERING↗
DOE OSTI2022

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

Towards reverse mode automatic differentiation of Kokkos-based codes

Derivative computation is a key component of optimization, sensitivity analysis, uncertainty quantification, and the solving of nonlinear problems. Automatic differentiation (AD) is a powerful technique for evaluating such derivatives, and in recent years, has been integrated into programming environments such as Jax, PyTorch, and TensorFlow to support derivative computations needed for training of machine learning models, facilitating wide-spread use of these technologies. The C++ language has become the de facto standard for scientific computing due to numerous factors, yet language complexity has made the wide-spread adoption of AD technologies for C++ difficult, hampering the incorporation of powerful differentiable programming approaches into C++ scientific simulations. This is exacerbated by the increasing emergence of architectures, such as GPUs, with limited memory capabilities and requiring massive thread-level concurrency. C++ AD tools must effectively use these environments to bring novel scientific simulations to next-generation DOE experimental and observational facilities. In this project, we investigated source transformation-based automatic differentiation using LLVM compiler infrastructure to automatically generate portable and efficient gradient computations of Kokkos-based code. We have demonstrated that our proposed strategy is feasible by investigating the usage of a prototype LLVM-based source transformation tool to generate gradients of simple functions made of sequences of simple Kokkos parallel regions. Speedups of up to 500x compared to Sacado were observed on NVIDIA V100 GPU.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

kynema-sgf [SWR-20-85]

Kynema-SGF (formerly AMR-Wind), wherein SGF stands for structured-grid fluid dynamics, is a massively parallel, block-structured adaptive-mesh, incompressible flow solver. The codebase was initiated in 2019 from incflo. The solver is built on top of the AMReX library. AMReX library provides the mesh data structures, mesh adaptivity, as well as the linear solvers used for solving the governing equations. Kynema-SGF is actively developed and maintained by a dedicated multi-institutional team from Lawrence Berkeley National Laboratory, National Laboratory of the Rockies, and Sandia National Laboratories. The primary applications for Kynema-SGF are: performing large-eddy simulations (LES) of atmospheric boundary layer (ABL) flows, simulating wind farm turbine-wake interactions using actuator disk or actuator line models for turbines, and as a background solver when coupled with a near-body solver (e.g., Kynema-UGF) with overset methodology to perform blade-resolved simulations of multiple wind turbines within a wind farm. For offshore applications, the ability to model the air-sea interaction effects and its impact on the ABL characteristics is another focus for the code development effort. As with other codes in the Kynema ecosystem, Kynema-SGF shares the following objectives: *an open, well-documented implementation of the state-of-the-art computational models for modeling wind farm flow physics at various fidelities that are backed by a comprehensive verification and validation (V&V) process; *be capable of performing the highest-fidelity simulations of flow fields within wind farms; and *be able to leverage the high-performance leadership class computing facilities available at DOE national laboratories.

Ananthan, Shreyas↗
DOE OSTI2021

Scale-dependent spatial variabilities of hydrological exchange flows and transit time in a large regulated river

Hydrological exchange flows (HEF) across the river-aquifer interface and the associated residence time of river water in the aquifer have important implications for contaminant plume migration and biogeochemical processes in the river corridor. HEFs and residence time are influenced by both subsurface physical features and hydrologic forcing related to the transport process, which can exhibit complex spatial and temporal variations. In this study, we used a massively parallel subsurface flow model and a particle-tracking model to study the influences of different control factors on spatial variability of HEFs and residence time distributions (RTD) in the Hanford Reach of the Columbia River in Washington State. A total number of 100M particles were randomly injected in time and space and then tracked in a model domain that covers a 51-km 2 area (15.1M model cells). We used hourly river stages and groundwater levels to drive the model to provide dynamic velocity fields for the particle tracking in the simulation period that was longer than 2 years. The groundwater flow simulation and particle-tracking results provide the first comprehensive assessment of the spatial distribution of HEFs and residence time in large complex river corridors. Overall, our results show that the aquifer hydrogeological structure has the strongest correlation with the extent and magnitude of exchange flux. The residence time exhibits complex patterns that are impacted by all the river geomorphologic, hydrodynamic, and hydrogeologic factors and are strongly correlated with the downwelling ratio of exchange flux. The new insights gained through this study can be used to support the development of reduced-order models of HEFs and RTDs for large complex river systems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Overview of ASDEX upgrade results in view of ITER and DEMO

Experiments on ASDEX Upgrade (AUG) in 2021 and 2022 have addressed a number of critical issues for ITER and EU DEMO. A major objective of the AUG programme is to shed light on the underlying physics of confinement, stability, and plasma exhaust in order to allow reliable extrapolation of results obtained on present day machines to these reactor-grade devices. Concerning pedestal physics, the mitigation of edge localised modes (ELMs) using resonant magnetic perturbations (RMPs) was found to be consistent with a reduction of the linear peeling-ballooning stability threshold due to the helical deformation of the plasma. Conversely, ELM suppression by RMPs is ascribed to an increased pedestal transport that keeps the plasma away from this boundary. Candidates for this increased transport are locally enhanced turbulence and a locked magnetic island in the pedestal. The enhanced D-alpha (EDA) and quasi-continuous exhaust (QCE) regimes have been established as promising ELM-free scenarios. Here, the pressure gradient at the foot of the H-mode pedestal is reduced by a quasi-coherent mode, consistent with violation of the high-n ballooning mode stability limit there. This is suggestive that the EDA and QCE regimes have a common underlying physics origin. In the area of transport physics, full radius models for both L- and H-modes have been developed. These models predict energy confinement in AUG better than the commonly used global scaling laws, representing a large step towards the goal of predictive capability. A new momentum transport analysis framework has been developed that provides access to the intrinsic torque in the plasma core. In the field of exhaust, the X-Point Radiator (XPR), a cold and dense plasma region on closed flux surfaces close to the X-point, was described by an analytical model that provides an understanding of its formation as well as its stability, i.e., the conditions under which it transitions into a deleterious MARFE with the potential to result in a disruptive termination. With the XPR close to the divertor target, a new detached divertor concept, the compact radiative divertor, was developed. Here, the exhaust power is radiated before reaching the target, allowing close proximity of the X-point to the target. No limitations by the shallow field line angle due to the large flux expansion were observed, and sufficient compression of neutral density was demonstrated. With respect to the pumping of non-recycling impurities, the divertor enrichment was found to mainly depend on the ionisation energy of the impurity under consideration. In the area of MHD physics, analysis of the hot plasma core motion in sawtooth crashes showed good agreement with nonlinear 2-fluid simulations. This indicates that the fast reconnection observed in these events is adequately described including the pressure gradient and the electron inertia in the parallel Ohm’s law. Concerning disruption physics, a shattered pellet injection system was installed in collaboration with the ITER International Organisation. Thanks to the ability to vary the shard size distribution independently of the injection velocity, as well as its impurity admixture, it was possible to tailor the current quench rate, which is an important requirement for future large devices such as ITER. Progress was also made modelling the force reduction of VDEs induced by massive gas injection on AUG. The H-mode density limit was characterised in terms of safe operational space with a newly developed active feedback control method that allowed the stability boundary to be probed several times within a single discharge without inducing a disruptive termination. Regarding integrated operation scenarios, the role of density peaking in the confinement of the ITER baseline scenario (high plasma current) was clarified. The usual energy confinement scaling ITER98(p,y) does not capture this effect, but the more recent H20 scaling does, highlighting again the importance of developing adequate physics based models. Advanced tokamak scenarios, aiming at large non-inductive current fraction due to non-standard profiles of the safety factor in combination with high normalised plasma pressure were studied with a focus on their access conditions. A method to guide the approach of the targeted safety factor profiles was developed, and the conditions for achieving good confinement were clarified. Based on this, two types of advanced scenarios (‘hybrid’ and ‘elevated’

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

A generalized and adaptable tensor-contraction-based cluster expansion formalism for multicomponent solids

Density functional theory (DFT)-based simulations of materials have first-principles accuracy, but are very computationally expensive. For simulating various properties of multi-component alloys, the cluster expansion (CE) technique has served as the standard workaround to improve computational efficiency. However, the standard CE technique is difficult to extend to exotic and/or low-symmetry lattices, often implemented via iteration over particular cluster types, which must be enumerated per lattice structure. In this work, we introduce the tensor cluster expansion (TCE), implemented in the open-source code tce-lib, which maps correlation functions to mixed tensor contractions, eliminating the need to iterate over cluster types and additionally making the calculation of correlation functions well-suited for massively parallel architectures like GPUs. We show that local interaction energies are an immediate consequence of the TCE formalism, yielding nearly $\mathcal{O}$(1) energy difference calculations. We then use this formalism to fit CE models for the TaW and CoNiCrFeMn systems, and use these models to respectively compute the enthalpy of mixing curve and Cowley short-range order parameters, showing excellent agreement with ground truth data.

Cluster expansion↗
DOE OSTI2021

Progress on Optimizing Wind Farms and Rotor Designs Using Adjoints

Modern wind plants are increasingly tasked with multiple performance objectives. In addition to designing plants that maximize power output and minimize the levelized cost of energy (LCOE), the design and operation of wind plants is increasingly influenced by challenges regarding grid integration of variable generation renewables. This places a growing emphasis on making wind plants more controllable and predictable. WindSE is a Reynolds-averaged Navier-Stokes (RANS) model designed around analytical gradient and adjoint methods, with the ability to capture terrain-induced effects, as shown in Figure 1. The recent addition of an unsteady solver with an actuator line method (ALM) and ongoing work to enable massively parallel optimizations gives it a unique niche to explore coupled plant-level controls and design problems. This code is an open source python package built on the FEniCS framework that utilizes fast, parallel PETSc solvers to model fluid flow throughout wind-farm scale domains. Two recent studies performed using WindSE demonstrate the capability to optimize under a wide variety of flow conditions and objective functions. In the first, we present an optimization focused on modifying the layout of a wind farm with a fixed number of turbines for maximum total power output [1]. This study highlights the ability to quickly perform simulations using the steady Navier-Stokes solver combined with rotors represented as actuator disks while also stressing the importance of capturing terrain-induced effects. Gradient-based optimization using the RANS equations is viable due to the inclusion of efficiently computed adjoint derivatives. We interpret the physical results of the optimal layout and also discuss the computational cost of scaling to larger problems. In the second study, we present the capabilities of the unsteady Navier-Stokes solver, where rotor-blade profiles represented by actuator lines are optimized to enhance wake steering effects and overall power production [2]. We quantify the wind plant performance gains obtained from this type of simultaneous control co-design optimization as compared to optimizing the blade design and yaw independently. Figure 2 shows the differences between a baseline two-turbine system and an optimized system where we fine-tune the blade chord profile. Results and challenges from each study are quickly summarized and used to motivate the current development efforts within WindSE. Current and future work is focused on enabling higher-resolution studies with more degrees of freedom through parallelization of both the simulation and optimization algorithms. We present benchmarking results to show that WindSE performs well in both weak- and strong-scaling tests and further demonstrate that the optimizer obtains the same convergence rates in both shared- and distributed-memory environments. Using larger wind farms, we can study deep-array effects within an optimization context, allowing the use of objective functions that have been previously unstudied. As an example, we present ongoing work on a blockage metric which characterizes the loss of available kinetic energy due to wake effects from multiple upstream turbines.

adjoint optimization↗
DOE OSTI2025

Broadband unidirectional visible imaging using wafer-scale nano-fabrication of multi-layer diffractive optical processors

We present a broadband and polarization-insensitive unidirectional imager that operates at the visible part of the spectrum, where image formation occurs in one direction, while in the opposite direction, it is blocked. This approach is enabled by deep learning-driven diffractive optical design with wafer-scale nano-fabrication using high-purity fused silica to ensure optical transparency and thermal stability. Our design achieves unidirectional imaging across three visible wavelengths (covering red, green, and blue parts of the spectrum), and we experimentally validated this broadband unidirectional imager by creating high-fidelity images in the forward direction and generating weak, distorted output patterns in the backward direction, in alignment with our numerical simulations. This work demonstrates wafer-scale production of diffractive optical processors, featuring 16 levels of nanoscale phase features distributed across two axially aligned diffractive layers for visible unidirectional imaging. This approach facilitates mass-scale production of ~0.5 billion nanoscale phase features per wafer, supporting high-throughput manufacturing of hundreds to thousands of multi-layer diffractive processors suitable for large apertures and parallel processing of multiple tasks. Beyond broadband unidirectional imaging in the visible spectrum, this study establishes a pathway for artificial-intelligence-enabled diffractive optics with versatile applications, signaling a new era in optical device functionality with industrial-level, massively scalable fabrication.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A massively parallel time-domain coupled electrodynamics–micromagnetics solver

We present a high-performance coupled electrodynamics–micromagnetics solver for full physical modeling of signals in microelectronic circuitry. The overall strategy couples a finite-difference time-domain approach for Maxwell’s equations to a magnetization model described by the Landau–Lifshitz–Gilbert equation. The algorithm is implemented in the Exascale Computing Project software framework, AMReX, which provides effective scalability on manycore and GPU-based supercomputing architectures. Furthermore, the code leverages ongoing developments of the Exascale Application Code, WarpX, which is primarily being developed for plasma wakefield accelerator modeling. Our temporal coupling scheme provides second-order accuracy in space and time by combining the integration steps for the magnetic field and magnetization into an iterative sub-step that includes a trapezoidal temporal discretization for the magnetization. The performance of the algorithm is demonstrated by the excellent scaling results on NERSC multicore and GPU systems, with a significant (59×) speedup on the GPU using a node-by-node comparison. We demonstrate the utility of our code by performing simulations of an electromagnetic waveguide and a magnetically tunable filter.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2019 Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory's (ORNL's) Leadership Computing Facility (OLCF) continues to surpass its operational target goals: supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high performance computing (HPC) needs; and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2019 was a big year as OLCF staff ran five world-class resources (the leadershipclass computers Titan and Summit, the large analysis cluster called Eos, and the massive parallel filesystems called Atlas and Alpine)) and also began power and cooling upgrades for a 2021 exascale system called Frontier. While continuing exceptional operation of Titan, Eos, and Rhea, the OLCF released the Summit supercomputer for production on January 1, 2019. Summit debuted as the most capable and efficient system in its class and has been recognized as the most powerful system in the world for its performance on both the high performance linpack (HPL) and conjugate gradient (HPCG) benchmark applications since June 2018 according to TOP500. Summit represents the culmination of a multiyear effort between the OLCF, IBM, NVIDIA, and Mellanox to deliver a system that is unmatched for modeling, simulation, data analysis, and learning. To hit the ground running with science-ready applications on day one, application teams worked closely with the OLCF through the Center for Accelerated Application Readiness (CAAR) program for years in advance of the Summit deployment. CY 2019 was filled with outstanding results and accomplishments: a very high rating from users on overall satisfaction for the sixth year in a row; a tremendous amount of core-hours delivered to researchers from two leadership-class systems; and success in delivering on the allocation split of roughly 60%, 30%, and 10% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director's Discretionary (DD) programs, respectively (see Operational Performance section). These accomplishments, coupled with the high utilization rates (overall and capability usage), represent the fulfillment of the promise of both leadership-class machines: efficient facilitation of leadership-class computational applications. Table ES.1 presents a summary of the 2019 OLCF metric targets and the associated results. More information can be found in the Operational Performance section for each OLCF resource. The scientific accomplishments of OLCF users are a strong indication of long-term operational success, with publications this year in such notable journals and publications as Nature, Nature Physics, Nature Plants, Physical Review X, Journal of the American Physical Society, Cell, Nano Letters, and Trends in Biotechnology. Crucial domain-specific discoveries facilitated by resources at the OLCF are described in the High Performance Computing Facility Operational Assessment 2019 Oak Ridge Leadership Computing Facility (OAR) Strategic Results section. For example, researchers used Summit to pinpoint and understand the production of proteins from genetic information, including mutations and the functional expression of disease (Section 8.2).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Green's function methods for multiphysics simulations (Final Report)

Green's functions are important tools for analyzing mathematical properties of partial differential equations (PDEs), and for numerically solving PDEs, especially when equations for the same operator but with multiple right hand sides need to be solved simultaneously. This proposal aims at developing efficient and accurate numerical methods for computing Green's functions, which can be used to tackle a challenging question in multiphysics simulation of DOE-mission science problems: how to couple quantum physics with classical physics. The key mathematical difficulty is properly formulate a "boundary condition" for the region described by quantum physics, and conventional approaches often model such boundary conditions in an empirical way. The proposed Green's function methods use the Dirichlet-to-Neumann map to formulate a boundary condition that is non-empirical and can couple the quantum and classical regions in an in principle exact way. The key ingredient of the new methods is to construct the Dirichlet-to-Neumann map in an efficient, accurate and versatile manner. The new methods have provably low complexity and are ideally suited for massively parallel and emerging many-core computational systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Collective mode Brownian dynamics: A method for fast relaxation of statistical ensembles

Sampling equilibrium configurations of correlated systems of particles with long relaxation times (e.g., polymeric solutions) using conventional molecular dynamics and Monte Carlo methods can be challenging. This is especially true for systems with complicated, extended bond network topologies and other interactions that make the use and design of specialized relaxation protocols infeasible. We introduce a method based on Brownian dynamics simulations that can reduce the computational time it takes to reach equilibrium and draw decorrelated samples. Importantly, the method is completely agnostic to the particle configuration and the specifics of interparticle forces. In particular, we develop a mobility matrix that excites non-local, collective motion of N particles and can be computed efficiently in $\mathcal{O}(N)$ time. Particle motion in this scheme is computed by integrating the overdamped Langevin equation with an Euler–Maruyama scheme, in which Brownian displacements are drawn efficiently using a low-rank representation of the mobility matrix in position and wave space. We demonstrate the efficacy of the method with various examples from the realm of soft condensed matter and release a massively parallel implementation of the code as a plugin for the open-source package HOOMD-blue [J. A. Anderson et al., J. Comput. Phys. 227, 5342 (2008) and J. Glaser et al., Comput. Phys. Commun. 192, 97 (2015)] which runs on graphics processing units

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Containers for Massive Ensemble of I/O Bound Hierarchical Coupled Simulations

We present our experience using containers to scale up a massive ensemble of coupled I/O bound workloads on the NERSC Cori supercomputer. We describe the design of a hierarchical simulation structure using the Integrated Plasma Simulator (IPS) that enables the flexible execution of coupled simulations at the system, node, and core level using the same coupling abstraction and API. The hierarchical design allows for the node-level execution to be efficiently executed using containers while not impacting the structure of the simulation at the system level. We demonstrate the viability of the approach by presenting experimental results from applications in coupled fusion plasma simulations that illustrate the performance impact of using containers to deploy the node-level workloads, in conjunction with the user mountable XFS file systems to ameliorate the load on the Lustre parallel file system. We also present results from production runs showing the ability of the ensemble simulations to scale to hundreds of Cori Haswell nodes, with little or no overhead.

Elwasif, Wael↗
DOE OSTI2020

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Novel Solver Algorithms for Nearly Singular Linear Systems Arising in Combustion Modelling

Direct Numerical Simulations of realistic combustion devices are extremely challenging due to the wide separation of scales in the simulation, for example an internal combustion (IC) engine chamber, and the flame thickness of a high-pressure flame. The PeleLMeX solver uses adaptive mesh refinement (AMR) to evolve multi-species reacting flows in the low Mach number limit at the Exascale and relies on an embedded boundary (EB) approach to represent complex geometries. In that framework, the EB geometries often give rise to very small cut-cells along the boundary, which translate into extreme ill-conditioning of the pressure-projection, with eigenvalues that span 15-16 orders of magnitude. In this talk, we focus on the case of a typical IC piston bowl geometry for which we present on a novel approach towards solving these nearly singular linear systems with ILU-based, C-AMG smoothers on massively parallel architectures. In particular, we use scaling and equilibration algorithms to handle the non-normality of the upper triangular factors. This enables us to approximate the highly sequential triangular solve algorithm, embedded in the AMG smoothing-solve phase, with Jacobi iterations. This approximation can be written as a convergent Neumann series whose terms are composed of highly parallel sparse matrix vector multiplications. The result is an algorithm that substantially decreases setup and solve time, compared to state-of-the-art, for these challenging linear systems.

combustion modelling↗
DOE OSTI2022

Cabana: A Performance Portable Library for Particle-Based Simulations

Particle-based simulations are ubiquitous throughout many fields of computational science and engineering, spanning the atomistic level with molecular dynamics (MD), to mesoscale particle-in-cell (PIC) simulations for solid mechanics, device-scale modeling with PIC methods for plasma physics, and massive N-body cosmology simulations of galaxy structures, with many other methods in between (Hockney & Eastwood, 1989). While these methods use particles to represent significantly different entities with completely different physical models, many low-level details are shared including performant algorithms for short- and/or long-range particle interactions, multi-node particle communication patterns, and other data management tasks such as particle sorting and neighbor list construction. Cabana is a performance portable library for particle-based simulations, developed as part of the Co-Design Center for Particle Applications (CoPA) within the Exascale Computing Project (ECP) (Alexander et al., 2020). The CoPA project and its full development scope, including ECP partner applications, algorithm development, and similar software libraries for quantum MD, is described in (Mniszewski et al., 2021). Cabana uses the Kokkos library for on-node parallelism (Edwards et al., 2014; Trott et al., 2022), enabling simulation on multi-core CPU and GPU architectures, and MPI for GPU-aware, multi-node communication. Cabana provides particle simulation capabilities on almost all current Kokkos backends, including serial execution, OpenMP (including OpenMP-Target for GPUs), CUDA (NVIDIA GPUs), HIP (AMD GPUs), and SYCL (Intel GPUs), providing a clear path for the coming generation of accelerator-based exascale hardware. Cabana builds on Kokkos by providing new particle data structures and particle algorithms resulting in a similar execution policy-based, node-level programming model that is intended to be used in addition to the core Kokkos library within an application. Cabana is designed as an application and physics agnostic, but particle-specific toolkit which can either be used to generate a new application, or to be used as needed in existing applications at various levels of invasiveness including through interfaces that wrap user memory in existing data structures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Towards performance portability in the Spark astrophysical magnetohydrodynamics solver in the Flash-X simulation framework

Simulations of core-collapse supernovae, and other astrophysical phenomena, are quintessential extreme-scale computing challenges. For core-collapse supernova simulations to be carried out by the ExaStar project under the Exascale Computing Project umbrella, a robust, efficient, and state-of-the-art magnetohydrodynamics solver is a critical requirement. In Flash-X, the primary software instrument for ExaStar, a new magnetohydrodynamics solver has been designed and implemented from the ground up to achieve accuracy and efficiency for simulations of complex astrophysical flows. This new solver, dubbed Spark, uses high-order spatial reconstruction, Runge-Kutta time integration, and an efficient cell-centered approach to satisfying the divergence-free condition for the magnetic fields. Spark was written to be optimized for data locality in cache hierarchy of CPUs. Since data locality optimizations for cache hierarchy are not directly compatible with those of accelerators, we have taken the approach of using program synthesis to avoid massive amounts of code replication that would be necessary if we were to maintain two different versions of the solver. Our program synthesis relies on a simple key-dictionary approach, implemented in python, that enables us to assemble the version of the solver suitable for the target hardware from code fragments identified by specific keys. In this work, we describe the data locality optimizations of the solver for CPUs and accelerators and the program synthesis tools that enable this portability. We also detail the parallel performance of Spark for both CPUs and accelerators.

97 MATHEMATICS AND COMPUTING↗