Final Report for "Extending Chombo with PETSc"
Explore the source record for details and available documents.
adaptive mesh refinement, matrix assembly, mesh re↗
SEARCH · Search NASA
Results for “Chombo”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A multiphysics coupling framework for exascale simulation of fracture evolution in subsurface energy applications
Predicting the evolution of fractured media is challenging due to coupled thermal, hydrological, chemical and mechanical processes that occur over a broad range of spatial scales, from the microscopic pore scale to field scale. We present a software framework and scientific workflow that couples the pore scale flow and reactive transport simulator Chombo-Crunch with the field scale geomechanics solver in GEOS to simulate fracture evolution in subsurface fluid-rock systems. This new multiphysics coupling capability comprises several novel features. An HDF5 data schema for coupling fracture positions between the two codes is employed and leverages the coarse resolution of the GEOS mechanics solver which limits the size of data coupled, and is, thus, not taxed by data resulting from the high resolution pore scale Chombo-Crunch solver. The coupling framework requires tracking of both before and after coarse nodal positions in GEOS as well as the resolved embedded boundary in Chombo-Crunch. We accomplished this by developing an approach to geometry generation that tracks the fracture interface between the two different methodologies. The GEOS quadrilateral mesh is converted to triangles which are organized into bins and an accessible tree structure; the nodes are then mapped to the Chombo representation using a continuous signed distance function that determines locations inside, on and outside of the fracture boundary. The GEOS positions are retained in memory on the Chombo-Crunch side of the coupling. The time stepping cadence for coupled multiphysics processes of flow, transport, reactions and mechanics is stable and demonstrates temporal reach to experimental time scales. The approach is validated by demonstration of 9 days of simulated time of a core flood experiment with fracture aperture evolution due to invasion of carbonated brine in wellbore-cement and sandstone. We also demonstrate usage of exascale computing resources by simulating a high resolution version of the validation problem on OLCF Frontier.
97 MATHEMATICS AND COMPUTING↗
Introducing Enabling Computational Tools to the Climate Sciences: Multi-Resolution Climate Modeling with Adaptive Cubed-Sphere Grids
The research investigates and advances strategies how to bridge the scale discrepancies between local, regional and global phenomena in climate models without the prohibitive computational costs of global cloud-resolving simulations. In particular, the research explores new frontiers in computational geoscience by introducing high-order Adaptive Mesh Refinement (AMR) techniques into climate research. AMR and statically-adapted variable-resolution approaches represent an emerging trend for atmospheric models and are likely to become the new norm in future-generation weather and climate models. The research advances the understanding of multi-scale interactions in the climate system and showcases a pathway how to model these interactions effectively with advanced computational tools, like the Chombo AMR library developed at the Lawrence Berkeley National Laboratory. The research is interdisciplinary and combines applied mathematics, scientific computing and the atmospheric sciences. In this research project, a hierarchy of high-order atmospheric models on cubed-sphere computational grids have been developed that serve as an algorithmic prototype for the finite-volume solution-adaptive Chombo-AMR approach. The foci of the investigations have lied on the characteristics of both static mesh adaptations and dynamically-adaptive grids that can capture flow fields of interest like tropical cyclones. Six research themes have been chosen. These are (1) the introduction of adaptive mesh refinement techniques into the climate sciences, (2) advanced algorithms for nonhydrostatic atmospheric dynamical cores, (3) an assessment of the interplay between resolved-scale dynamical motions and subgrid-scale physical parameterizations, (4) evaluation techniques for atmospheric model hierarchies, (5) the comparison of AMR refinement strategies and (6) tropical cyclone studies with a focus on multi-scale interactions and variable-resolution modeling. The results of this research project demonstrate significant advances in all six research areas. The major conclusions are that statically-adaptive variable-resolution modeling is currently becoming mature in the climate sciences, and that AMR holds outstanding promise for future-generation weather and climate models on high-performance computing architectures.
58 GEOSCIENCES↗
A space-time parallel algorithm with adaptive mesh refinement for computational fluid dynamics
This work describes a space-time parallel algorithm with space-time adaptive mesh refinement (AMR). AMR with subcycling is added to multigrid reduction-in-time (MGRIT) in order to provide solution efficient adaptive grids with a reduction in work performed on coarser grids. This algorithm is achieved by integrating two software libraries: XBraid (Parallel time integration with multigrid. https://computation.llnl.gov/projects/parallel-timeintegration-multigrid) and Chombo (Chombo software package for AMR applications—design document, 2014). The former is a parallel time integration library using multigrid and the latter is a massively parallel structured AMR library. Employing this adaptive space-time parallel algorithm is Chord (Comput Fluids 123:202–217, 2015), a computational fluid dynamics (CFD) application code for solving compressible fluid dynamics problems. For the same solution accuracy, speedups are demonstrated from the use of space-time parallelization over the time-sequential integration on Couette flow and Stokes’ second problem. On a transient Couette flow case, at least a 1.5× speedup is achieved, and with a time periodic problem, a speedup of up to 13.7× over the time-sequential case is obtained. In both cases, the speedup is achieved by adding processors and exploring additional parallelization in time. The numerical experiments show the algorithm is promising for CFD applications that can take advantage of the time parallelism. Future work will focus on improving the parallel performance and providing more tests with complex fluid dynamics to demonstrate the full potential of the algorithm.
97 MATHEMATICS AND COMPUTING↗
Composite matrix construction for structured grid adaptive mesh refinement
Structured-grid adaptive mesh refinement (SAMR) is an approach to mesh generation that supports structured access to data and adaptive mesh refinement for discretized partial differential equations (PDEs). Solution algorithms often require that an inverse of an operator be applied, a system of algebraic equations must be solved, and this process is often the primary computational cost in an application. SAMR is well suited to geometric multigrid solvers, which can be effective, but often do not adapt well to complex geometry including material coefficients. Algebraic multigrid (AMG) is more robust in the face of complex geometry, in both boundary conditions and internal material interfaces. AMG requires a stored matrix linearization of the operator. We discuss an approach, and an implementation in the Chombo block-structured AMR framework, for constructing composite grid matrices from a SAMR hierarchy of grids for use in linear solvers in the PETSc numerical library. We consider a case study with the Chombo-based BISICLES ice sheet modeling application.
97 MATHEMATICS AND COMPUTING↗
Versioned distributed arrays for resilience in scientific applications: Global view resilience
Exascale studies project reliability challenges for future high-performance computing (HPC) systems. We propose the Global View Resilience (GVR) system, a library that enables applications to add resilience in a portable, application-controlled fashion using versioned distributed arrays. We describe GVR’s interfaces to distributed arrays, versioning, and cross-layer error recovery. Using several large applications (OpenMC, the preconditioned conjugate gradient solver PCG, ddcMD, and Chombo), we evaluate the programmer effort to add resilience. The required changes are small (<2% LOC), localized, and machine-independent, requiring no software architecture changes. We also measure the overhead of adding GVR versioning and show that generally overheads <2% are achieved. We conclude that GVR’s interfaces and implementation are flexible and portable and create a gentle-slope path to tolerate growing error rates in future systems.
97 MATHEMATICS AND COMPUTING↗
Approaches for the Simulation of Coupled Processes in Evolving Fractured Porous Media Enabled by Exascale Computing
Models have historically represented fractured porous media with continuum descriptions that characterize the media using bulk parameters. The impact of small-scale features is not captured in these models, although they may be controlling the performance of subsurface applications. Pore-scale models can simulate processes in small-scale features by representing the pore space geometry explicitly but are computationally expensive for large domains. The alternative multiscale approach entails the combination of pore-scale and continuum-scale descriptions in a single framework. We use Chombo-Crunch, a computational capability that discretizes complex geometries with an adaptive, embedded boundary method to contrast these two approaches. Chombo-Crunch takes advantage of recent computational performance and memory bandwidth improvements resulting from the emergence of exascale computing resources. These combined improvements enable the efficient simulation of reactive transport in fractured media with a high degree of fidelity and the ability to capture the control small-scale processes exert on the overall medium evolution.
42 ENGINEERING↗
A survey of high level frameworks in block-structured adaptive mesh refinement packages
Over the last decade block-structured adaptive mesh refinement (SAMR) has found increasing use in large, publicly available codes and frameworks. SAMR frameworks have evolved along different paths. Some have stayed focused on specific domain areas, others have pursued a more general functionality, providing the building blocks for a larger variety of applications. In this survey paper we examine a representative set of SAMR packages and SAMR-based codes that have been in existence for half a decade or more, have a reasonably sized and active user base outside of their home institutions, and are publicly available. The set consists of a mix of SAMR packages and application codes that cover a broad range of scientific domains. We look at their high-level frameworks, their design trade-offs and their approach to dealing with the advent of radical changes in hardware architecture. Finally, the codes included in this survey are BoxLib, Cactus, Chombo, Enzo, FLASH, and Uintah.
97 MATHEMATICS AND COMPUTING↗
Simulation of mineral dissolution at the pore scale with evolving fluid-solid interfaces: review of approaches and benchmark problem set
This manuscript presents a benchmark problem for the simulation of single-phase flow, reactive transport, and solid geometry evolution at the pore scale. The problem is organized in three parts that focus on specific aspects: flow and reactive transport (part I), dissolution-driven geometry evolution in two dimensions (part II), and an experimental validation of three-dimensional dissolution-driven geometry evolution (part III). Five codes are used to obtain the solution to this benchmark problem, including Chombo-Crunch, OpenFOAM-DBS, a lattice Boltzman code, Vortex, and dissolFoam. These codes cover a good portion of the wide range of approaches typically employed for solving pore-scale problems in the literature, including discretization methods, characterization of the fluid-solid interfaces, and methods to move these interfaces as a result of fluid-solid reactions. A short review of these approaches is given in relation to selected published studies. Results from the simulations performed by the five codes show remarkable agreement both quantitatively--based on upscaled parameters such as surface area, solid volume, and effective reaction rate--and qualitatively--based on comparisons of shape evolution. This outcome is especially notable given the disparity of approaches used by the codes. Therefore, these results establish a strong benchmark for the validation and testing of pore-scale codes developed for the simulation of flow and reactive transport with evolving geometries. They also underscore the significant advances seen in the last decade in tools and approaches for simulating this type of problem.
58 GEOSCIENCES↗
Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.
Computer Science↗
Analyzing the Adaptive Mesh Refinement (AMR) Characteristics of a High-Order 2D Cubed-Sphere Shallow-Water Model
Adaptive mesh refinement (AMR) is a technique that has been featured only sporadically in atmospheric science literature. This study aims to demonstrate the utility of AMR for simulating atmospheric flows. Several test cases are implemented in a 2D shallow-water model on the sphere using the Chombo-AMR dynamical core. This high-order finite-volume model implements adaptive refinement in both space and time on a cubed-sphere grid using a mapped-multiblock mesh technique. The tests consist of the passive advection of a tracer around moving vortices, a steady-state geostrophic flow, an unsteady solid-body rotation, a gravity wave impinging on a mountain, and the interaction of binary vortices. Both static and dynamic refinements are analyzed to determine the strengths and weaknesses of AMR in both complex flows with small-scale features and large-scale smooth flows. The different test cases required different AMR criteria, such as vorticity or height-gradient based thresholds, in order to achieve the best accuracy for cost. The simulations show that the model can accurately resolve key local features without requiring global high-resolution grids. The adaptive grids are able to track features of interest reliably without inducing noise or visible distortions at the coarse–fine interfaces. Finally and furthermore, the AMR grids keep any degradations of the large-scale smooth flows to a minimum.
54 ENVIRONMENTAL SCIENCES↗
Exploring versioned distributed arrays for resilience in scientific applications: Global view resilience
Exascale studies project reliability challenges for future HPC systems. We present the Global View Resilience (GVR) system, a library for portable resilience. GVR begins with a subset of the Global Arrays interface, and adds new capabilities to create versions, name versions, and compute on version data. Applications can focus versioning where and when it is most productive, and customize for each application structure independently. This control is portable, and its embedding in application source makes it natural to express and easy to maintain. The ability to name multiple versions and “partially materialize” them efficiently makes ambitious forward-recovery based on “data slices” across versions or data structures both easy to express and efficient. Using several large applications (OpenMC, preconditioned conjugate gradient (PCG) solver, ddcMD, and Chombo), we evaluate the programming effort to add resilience. The required changes are small (< 2% lines of code (LOC)), localized and machine-independent, and perhaps most important, require no software architecture changes. We also measure the overhead of adding GVR versioning and show that overheads < 2% are generally achieved. This overhead suggests that GVR can be implemented in large-scale codes and support portable error recovery with modest investment and runtime impact. Our results are drawn from both IBM BG/Q and Cray XC30 experiments, demonstrating portability. We also present two case studies of flexible error recovery, illustrating how GVR can be used for multi-version rollback recovery, and several different forward-recovery schemes. GVR’s multi-version enables applications to survive latent errors (silent data corruption) with significant detection latency, and forward recovery can make that recovery extremely efficient. Lastly, our results suggest that GVR is scalable, portable, and efficient. GVR interfaces are flexible, supporting a variety of recovery schemes, and altogether GVR embodies a gentle-slope path to tolerate growing error rates in future extreme-scale systems.
97 MATHEMATICS AND COMPUTING↗
Collaborative Research: A Model of Partially Ionized Plasma Flows with Kinetic Treatment of Neutral Atoms and Nonthermal Ions
Interactions of flows of partially ionized, magnetized plasma are frequently accompanied by the presence of both thermal and non-thermal (pickup) ion components. Such interactions cannot be modeled using traditional MHD equations and require more advanced approaches to treat them. If a nonthermal component of ions is formed due to charge exchange and collisions between the thermal (core) ions and neutrals, it experiences the action of magnetic field, its distribution function is isotropized, and it soon acquires the velocity of the ambient plasma without being thermodynamically equilibrated. This situation, e. g., takes place in the outer heliosphere –- the part of interstellar space beyond the solar system whose properties are determined by the solar wind interaction with the local interstellar medium. This is also possible in laboratory, at million degrees and above, when plasma is conducting electricity far too well, which makes Ohmic heating ineffective. To attain the target temperatures one needs additional heating eventually playing a dominant role. Among such sources is a so-called neutral particle beam heating. This is a wide-spread technique (Joint European Torus and International Thermonuclear Experimental Reactor experiments) based on the injection of powerful beams of neutral atoms into ohmically preheated plasma. In this project we have investigated the energy and density separation between the thermal and nonthermal components in the solar wind and interstellar plasmas. A new model has been developed in which we solve the ideal MHD equations for mixture of all ions and the kinetic Boltzmann equation to describe the transport of neutral atoms. As a separate capability, we can treat the flow of neutral atoms in a multi-component fashion, where neutral atoms born in each thermodynamically distinct region are governed by the Euler gas dynamic equations. We also describe the behavior of pickup ions either kinetically, using the Fokker--Planck equation, or as a separate fluid. Our numerical simulations have demonstrated that pickup ions play a major role in the interaction of the solar wind and (partially ionized) interstellar medium plasmas. Our teams have investigated the stability of the surface (the heliopause) that separates the solar wind from the local interstellar medium, the transport of galactic cosmic rays, the properties of the heliotail flow, and modifications to the bow wave in front of the heliopause due to charge exchange between the neutral H atoms born in the solar wind and interstellar ions. Modeling results have been validated against observational data, such as obtained by the Interstellar Boundary Explorer (IBEX), and made it possible to shed light on the structure of energetic neutral atom maps created by this spacecraft.. We have also demonstrated that charge-exchange modulated heliosphere is a source of anisotropy of the multi-TeV cosmic ray flux observed in a number of Earth-bound air shower experiments. Newly developed codes are implemented within a Multi-Scale Fluid-Kinetic Simulation Suite (MS-FLUKSS), a publicly available code being developed by our team for over 12 years. MS-FLUKSS scales well up to 160,000 computing cores and has been ported on major supercomputers in the country. Efficient parallelization and data choreography in the continuum simulation modules are provided by Chombo, an adaptive mesh refinement framework managed by Phillip Colella's team at LBNL. We have implemented in-house, hybrid (MPI+OpenMP) parallelization of the kinetic modules that solve the Boltzmann equation with a Monte Carlo method. Currently, the kinetic modules are being rewritten to take advantage of the modern CPU-GPU supercomputer architecture. The scope of the project allowed us to enhance plasma research and education in such broad, multidisciplinary field as physics of partially ionized plasma and its application to space physics and fusion science. Besides the impact on the modeling of complex physical systems, our approach to computational resource management for complex codes utilizing multiple algorithm technologies appears to be a major advance on current approaches. The development of sophisticated resource management will be essential for all future modeling efforts that incorporate a diversity of scales and physical processes. Our effort provided leadership in promoting computational science and plasma physics within the UAH and FIT campuses and, through the training of a broad spectrum of scientists and engineers, fostering new technologies across the country.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
Collaborative Research: A Model of Partially Ionized Plasma Flows with Kinetic Treatment of Neutral Atoms and Nonthermal Ions
Interactions of flows of partially ionized, magnetized plasma are frequently accompanied by the presence of both thermal and non-thermal (pickup) ion components. Such interactions cannot be modeled using traditional MHD equations and require more advanced approaches to treat them. If a nonthermal component of ions is formed due to charge exchange and collisions between the thermal (core) ions and neutrals, it experiences the action of magnetic field, its distribution function is isotropized, and it soon acquires the velocity of the ambient plasma without being thermodynamically equilibrated. This situation, e. g., takes place in the outer heliosphere - the part of interstellar space beyond the solar system whose properties are determined by the solar wind interaction with the local interstellar medium. This is also possible in laboratory, at million degrees and above, when plasma is conducting electricity far too well, which makes Ohmic heating ineffective. To attain the target temperatures one needs additional heating eventually playing a dominant role. Among such sources is a so-called neutral particle beam heating. This is a wide-spread technique (Joint European Torus and International Thermonuclear Experimental Reactor experiments) based on the injection of powerful beams of neutral atoms into ohmically preheated plasma. In this project we have investigated the energy and density separation between the thermal and nonthermal components in the solar wind and interstellar plasmas. A new model has been developed in which we solve the ideal MHD equations for mixture of all ions and the kinetic Boltzmann equation to describe the transport of neutral atoms. As a separate capability, we can treat the flow of neutral atoms in a multi-component fashion, where neutral atoms born in each thermodynamically distinct regions are governed by the Euler gas dynamic equations. We also describe the behavior of pickup ions either kinetically, using the Fokker–Planck equation, or as a separate fluid. Our numerical simulations have demonstrated that pickup ions play a major role in the interaction of the solar wind and (partially ionized) interstellar medium plasmas. Our teams have investigated the stability of the surface (the heliopause) that separates the solar wind from the local interstellar medium, the transport of galactic cosmic rays, the properties of the heliotail flow, and modifications to the bow wave in front of the heliopause due to charge exchange between the neutral H atoms born in the solar wind and interstellar ions. Modeling results have been validated against observational data, such as obtained by the Interstellar Boundary Explorer (IBEX), and made it possible to shed light on the structure of energetic neutral atom maps created by this spacecraft.. We have also demonstrated that charge-exchange modulated heliosphere is a source of anisotropy of the multi-TeV cosmic ray flux observed in a number of Earth-bound air shower experiments. Newly developed codes are implemented within a Multi-Scale Fluid-Kinetic Simulation Suite (MS-FLUKSS), a publicly available code being developed by our team for over 12 years. MS-FLUKSS scales well up to 160,000 computing cores and has been ported on major supercomputers in the country. Efficient parallelization and data choreography in the continuum simulation modules are provided by Chombo, an adaptive mesh refinement framework managed by Phillip Colella’s team at LBNL. We have implemented in-house, hybrid (MPI+OpenMP) parallelization of the kinetic modules that solve the Boltzmann equation with a Monte Carlo method. Currently, the kinetic modules are being rewritten to take advantage of the modern CPU-GPU supercomputer architecture. The scope of the project allowed us to enhance plasma research and education in such broad, multidis- ciplinary field as physics of partially ionized plasma and its application to space physics and fusion science. Besides the impact on the modeling of complex physical systems, our approach to computational resource management for complex codes utilizing multiple algorithm technologies appears to be a major advance on current approaches. The development of sophisticated resource management will be essential for all future modeling efforts that incorporate a diversity of scales and physical processes. Our effort provided leadership in promoting computational science and plasma physics within the UAH and FIT campuses and, through the training of a broad spectrum of scientists and engineers, foster new technologies across the country.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
SUHMO: an adaptive mesh refinement SUbglacial Hydrology MOdel v1.0
Abstract. Water flowing under ice sheets and glaciers can have a strong influence on ice dynamics, particularly through pressure changes, suggesting that a comprehensive ice sheet model should include the effect of basal hydrology. Modeling subglacial hydrology remains a challenge, however, mainly due to the range of spatial and temporal scales involved – from subglacial channels to vast subglacial lakes. Additionally, networks of subglacial drainage channels dynamically evolve over time. To address some of these challenges, we have developed an adaptive mesh refinement (AMR) model based on the Chombo software framework. We extend the model proposed by Sommers et al. (2018) with a small but significant change to accommodate the transition from unresolved to resolved flow features. We handle the strong nonlinearities present in the equations by resorting to an efficient nonlinear full approximation scheme multigrid (FAS-MG) algorithm. We outline the details of the algorithm and present convergence analysis results demonstrating its good performance. Additionally, we present results validating our approach, using test cases from the Subglacial Hydrology Model Intercomparison Project (SHMIP) (de Fleurian et al., 2018). We finish by presenting a more complex, 100 km-by-100 km synthetic test case with peaks and valleys that we use to investigate the effective pressure distribution as the number of AMR levels increases. These preliminary results suggest that a minimum spatial resolution is needed to properly capture channel features, but additional work is required to precisely quantify this and its impact on accurately modeling the coupled ice sheet–hydrology system. The efficiency of our approach, relying on localized refinement, is also demonstrated. Future work will include coupling the SUbglacial Hydrology MOdel (SUHMO) with the BISICLES AMR ice sheet model (Cornford et al., 2013), both built on the same numerical framework.
58 GEOSCIENCES↗