Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel mesh generation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

PHISICS/RELAP5-3D RESULTS FOR EXERCISES II-1 AND II-2 OF THE OECD/NEA MHTGR-350 BENCHMARK

The Idaho National Laboratory (INL) Advanced Reactor Technologies (ART) High-Temperature Gas-Cooled Reactor (HTGR) Methods group currently leads the Modular High-Temperature Gas-Cooled Reactor (MHTGR) 350 benchmark. The benchmark consists of a set of lattice-depletion, steady-state, and transient problems that can be used by HTGR simulation groups to assess the performance of their code suites. The paper summarizes the results obtained for the first two transient exercises defined for Phase II of the benchmark. The Parallel and Highly Innovative Simulation for INL Code System (PHISICS), coupled with the INL system code RELAP5-3D, was used to generate the results for the Depressurized Conduction Cooldown (DCC) (exercise II-1a) and Pressurized Conduction Cooldown (PCC) (exercise II-2) transients. These exercises require the time-dependent simulation of coupled neutronics and thermal-hydraulics phenomena, and utilize the steady-state solution previously obtained for exercise I-3 of Phase I. This paper also includes a comparison of the benchmark results obtained with a traditional system code “ring” model against a more detailed “block” model that include kinetics feedback on an individual block level and thermal feedbacks on a triangular sub-mesh. The higher spatial fidelity that can be obtained by the block model is illustrated with comparisons of the maximum fuel temperatures, especially in the case of natural convection conditions that dominate the DCC and PCC events. Differences up to 125 K (or 10%) were observed between the ring and block model predictions of the DCC transient, mostly due to the block model’s capability of tracking individual block decay powers and more detailed helium flow distributions. In general, the block model only required DCC and PCC calculation times twice as long as the ring models, and it therefore seems that the additional development and calculation time required for the block model could be worth the gain that can be obtained in the spatial resolution

Advanced Reactor Technologies↗

Voronoi Meshing to Accurately Capture Geological Structure in Subsurface Simulations

Mesh generation lies at the interface of geological modeling and reservoir simulation. Highly skewed or very small grid cells may be necessary to accurately capture the geometry of geological features, but the resulting poorly scaled or small grid cells can have a substantial negative impact on simulator accuracy and speed. One way to minimize numerical errors caused by gridding complex structures is to simulate on high-quality Voronoi meshes, which reduce grid orientation effects in fluid flow. This work presents a complete methodology to create Voronoi simulation grids, model fluid flow in complex geological systems, and visualize the results. A recently developed Voronoi meshing method that can automatically generate provably good unstructured meshes that conform to input surfaces creating closed volumes is used. Initially an analytical benchmark simulation is presented to validate the quality of the meshes and simulation results and demonstrate the superiority of simulation results using Voronoi meshes over flexed-hexahedral meshes on a domain with internal features. Next, meshes are created for test structures representing four of the most common geological features in the subsurface: layering, pinch-out, an interior lens that tapers to zero thickness on all sides and a fault with offset. Two benchmark flow simulations are run for each test structure. Finally, a realistic geological example for CO 2 injection into an anticline is simulated. Three realizations of the Voronoi mesh at the same resolution are generated for the simulations. Each mesh is highly refined near the injection wells and coarse in areas of less interest. These three meshes are used to model the CO 2 plume in the subsurface as it migrates to the top of the structure and then fills downward. Simulations on the meshes with randomly generated elements inside the input volumes each give slightly different fingering patterns for the viscous-unstable buoyant gas flow. The results presented in this work show a promising step towards utilizing fully automated Voronoi meshing for subsurface flow simulations in complex geology.

58 GEOSCIENCES↗

Enriched immersed finite element and isogeometric analysis: algorithms and data structures

Immersed finite element methods provide a convenient analysis framework for problems involving geometrically complex domains, such as those found in topology optimization and microstructures for engineered materials. However, their implementation remains a major challenge due to, among other things, the need to apply nontrivial stabilization schemes and generate custom quadrature rules. This article introduces the robust and computationally efficient algorithms and data structures comprising an immersed finite element preprocessing framework. The input to the preprocessor consists of a background mesh and one or more geometries defined on its domain. The output is structured into groups of elements with custom quadrature rules formatted such that common finite element assembly routines may be used without or with only minimal modifications. The key to the preprocessing framework is the construction of material topology information, concurrently with the generation of a quadrature rule, which is then used to perform enrichment and generate stabilization rules. While the algorithmic framework applies to a wide range of immersed finite element methods using different types of meshes, integration, and stabilization schemes, the preprocessor is presented within the context of the extended isogeometric analysis. This method utilizes a structured B-spline mesh, a generalized Heaviside enrichment strategy considering the material layout within individual basis functions’ supports, and face-oriented ghost stabilization. Using a set of examples, the effectiveness of the enrichment and stabilization strategies is demonstrated alongside the preprocessor’s robustness in geometric edge cases. Additionally, the performance and parallel scalability of the implementation are evaluated.

Computer implementation↗

SPARC: Demonstrate burst-buffer-based checkpoint/restart on ATS-1.

Recent high-performance computing (HPC) platforms such as the Trinity Advanced Technology System (ATS-1) feature burst buffer resources that can have a dramatic impact on an application’s I/O performance. While these non-volatile memory (NVM) resources provide a new tier in the storage hierarchy, developers must find the right way to incorporate the technology into their applications in order to reap the benefits. Similar to other laboratories, Sandia is actively investigating ways in which these resources can be incorporated into our existing libraries and workflows without burdening our application developers with excessive, platform-specific details. This FY18Q1 milestone summaries our progress in adapting the Sandia Parallel Aerodynamics and Reentry Code (SPARC) in Sandia’s ATDM program to leverage Trinity’s burst buffers for checkpoint/restart operations. We investigated four different approaches with varying tradeoffs in this work: (1) simply updating job script to use stage-in/stage out burst buffer directives, (2) modifying SPARC to use LANL’s hierarchical I/O (HIO) library to store/retrieve checkpoints, (3) updating Sandia’s IOSS library to incorporate the burst buffer in all meshing I/O operations, and (4) modifying SPARC to use our Kelpie distributed memory library to store/retrieve checkpoints. Team members were successful in generating initial implementation for all four approaches, but were unable to obtain performance numbers in time for this report (reasons: initial problem sizes were not large enough to stress I/O, and SPARC refactor will require changes to our code). When we presented our work to the SPARC team, they expressed the most interest in the second and third approaches. The HIO work was favored because it is lightweight, unobtrusive, and should be portable to ATS-2. The IOSS work is seen as a long-term solution, and is favored because all I/O work (including checkpoints) can be deferred to a single library.

97 MATHEMATICS AND COMPUTING↗

Application of the INSTANT-HPS PN Transport Code to the C5G7 Benchmark Problem

INSTANT is the INL's next generation neutron transport solver to support high-fidelity multi-physics reactor simulation INSTANT is in continuous development to extend its capability Code is designed to take full advantage of middle to large cluster (10-1000 processors) Code is designed to focus on method adaptation while also mesh adaptation will be possible. It utilizes the most modern computing techniques to generate a neutronics tool of full-core transport calculations for reactor analysis and design. It can perform calculations on unstructured 2D/3D triangular, hexagonal and Cartesian geometries. Calculations can be easily extended to more geometries because of the independent mesh framework coded with the model Fortran. This code has a multigroup solver with thermal rebalance and Chebyshev acceleration. It employs second-order PN and Hybrid Finite Element method (PNHFEM) discretization scheme. Three different in-group solvers - preconditioned Conjugate Gradient (CG) method, preconditioned Generalized Minimal Residual Method (GMRES) and Red-Black iteration - have been implemented and parallelized with the spatial domain decomposition in the code. The input is managed with extensible markup language (XML) format. 3D variables including the flux distributions are outputted into VTK files, which can be visualized by tools such as VisIt and ParaView. An extension of the code named INSTANTHPS provides the capability to perform 3D heterogeneous transport calculations within fuel pins. C5G7 is an OECD/NEA benchmark problem created to test the ability of modern deterministic transport methods and codes to treat reactor core problems without spatial homogenization. This benchmark problem had been widely analyzed with various code packages. In this transaction, results of the applying the INSTANT-HPS code to the C5G7 problem are summarized.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Quinoa, Version 0.2

Quinoa is a set of computational tools that enables research and numerical analysis in fluid dynamics. At this time it remains a test-bed to experiment with various algorithms using fully asynchronous runtime systems. Currently, Quinoa consists of the following tools: (1) Walker, a numerical integrator for systems of stochastic differential equations in time. It is a mathematical tool to analyze and design the behavior of stochastic differential equations. It allows the estimation of arbitrary coupled statistics and probability density functions and is currently used for the design of statistical moment approximations for multiple mixing materials in variable-density turbulence. (2) Inciter, an overdecomposition-aware finite element field solver for partial differential equations using 3D unstructured grids. Inciter is used to research asynchronous mesh-based algorithms and to experiment with coupling asynchronous to bulk-synchronous parallel code. Two planned new features of Inciter, compared to the previous release (LA-CC-16-015), to be implemented in 2017, are (a) a simple Navier-Stokes solver for ideal single-material compressible gases, and (b) solution-adaptive mesh refinement (AMR), which enables dynamically concentrating compute resources to regions with interesting physics. Using the NS-AMR problem we plan to explore how to scale such high-load-imbalance simulations, representative of large production multiphysics codes, to very large problems on very large computers using an asynchronous runtime system. (3) RNGTest, a test harness to subject random number generators to stringent statistical tests enabling quantitative ranking with respect to their quality and computational cost. (4) UnitTest, a unit test harness, running hundreds of tests per second, capable of testing serial, synchronous, and asynchronous functions. (5) MeshConv, a mesh file converter that can be used to convert 3D tetrahedron meshes from and to either of the following formats: Gmsh, http://www.geuz.org/gmsh, Netgen, http://sourceforge.net/apps/mediawiki/netgen-mesher, ExodusII, http://sourceforge.net/projects/exodusii, HyperMesh, http://www.altairhyperworks.com/product/HyperMesh.

Bakosi, Jozsef↗

Collision of Physics and Software in the Monte Carlo Application Toolkit (MCATK)

The topic is presented in a series of slides organized as follows: MCATK overview, development strategy, available algorithms, problem modeling (sources, geometry, data, tallies), parallelism, miscellaneous tools/features, example MCATK application, recent areas of research, and summary and future work. MCATK is a C++ component-based Monte Carlo neutron-gamma transport software library with continuous energy neutron and photon transport. Designed to build specialized applications and to provide new functionality in existing general-purpose Monte Carlo codes like MCNP, it reads ACE formatted nuclear data generated by NJOY. The motivation behind MCATK was to reduce costs. MCATK physics involves continuous energy neutron & gamma transport with multi-temperature treatment, static eigenvalue (k eff and α) algorithms, time-dependent algorithm, and fission chain algorithms. MCATK geometry includes mesh geometries and solid body geometries. MCATK provides verified, unit-test Monte Carlo components, flexibility in Monte Carlo application development, and numerous tools such as geometry and cross section plotters.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

A Novel Coarsening Method for Scalable and Efficient Mesh Generation

In this paper, we propose a novel mesh coarsening method called brick coarsening method. The proposed method can be used in conjunction with any graph partitioners and scales to very large meshes. This method reduces problem space by decomposing the original mesh into fixed-size blocks of nodes called bricks, layered in a similar way to conventional brick laying, and then assigning each node of the original mesh to appropriate brick. Our experiments indicate that the proposed method scales to very large meshes while allowing simple RCB partitioner to produce higher-quality partitions with significantly less edge cuts. Our results further indicate that the proposed brick-coarsening method allows more complicated partitioners like PT-Scotch to scale to very large problem size while still maintaining good partitioning performance with relatively good edge-cut metric. Graph partitioning is an important problem that has many scientific and engineering applications in such areas as VLSI design, scientific computing, and resource management. Given a graph G = (V,E), where V is the set of vertices and E is the set of edges, (k-way) graph partitioning problem is to partition the vertices of the graph (V) into k disjoint groups such that each group contains roughly equal number of vertices and the number of edges connecting vertices in different groups is minimized. Graph partitioning plays a key role in large scientific computing, especially in mesh-based computations, as it is used as a tool to minimize the volume of communication and to ensure well-balanced load across computing nodes. The impact of graph partitioning on the reduction of communication can be easily seen, for example, in different iterative methods to solve a sparse system of linear equation. Here, a graph partitioning technique is applied to the matrix, which is basically a graph in which each edge is a non-zero entry in the matrix, to allocate groups of vertices to processors in such a way that many of matrix-vector multiplication can be performed locally on each processor and hence to minimize communication. Furthermore, a good graph partitioning scheme ensures the equal amount of computation performed on each processor. Graph partitioning is a well known NP-complete problem, and thus the most commonly used graph partitioning algorithms employ some forms of heuristics. These algorithms vary in terms of their complexity, partition generation time, and the quality of partitions, and they tend to trade off these factors. A significant challenge we are currently facing at the Lawrence Livermore National Laboratory is how to partition very large meshes on massive-size distributed memory machines like IBM BlueGene/P, where scalability becomes a big issue. For example, we have found that the ParMetis, a very popular graph partitioning tool, can only scale to 16K processors. An ideal graph partitioning method on such an environment should be fast and scale to very large meshes, while producing high quality partitions. This is an extremely challenging task, as to scale to that level, the partitioning algorithm should be simple and be able to produce partitions that minimize inter-processor communications and balance the load imposed on the processors. Our goals in this work are two-fold: (1) To develop a new scalable graph partitioning method with good load balancing and communication reduction capability. (2) To study the performance of the proposed partitioning method on very large parallel machines using actual data sets and compare the performance to that of existing methods. The proposed method achieves the desired scalability by reducing the mesh size. For this, it coarsens an input mesh into a smaller size mesh by coalescing the vertices and edges of the original mesh into a set of mega-vertices and mega-edges. A new coarsening method called brick algorithm is developed in this research. In the brick algorithm, the zones in a given mesh are first grouped into fixed size blocks called bricks. These brick are then laid in a way similar to conventional brick laying technique, which reduces the number of neighboring blocks each block needs to communicate. Contributions of this research are as follows: (1) We have developed a novel method that scales to a really large problem size while producing high quality mesh partitions; (2) We measured the performance and scalability of the proposed method on a machine of massive size using a set of actual large complex data sets, where we have scaled to a mesh with 110 million zones using our method. To the best of our knowledge, this is the largest complex mesh that a partitioning method is successfully applied to; and (3) We have shown that proposed method can reduce the number of edge cuts by as much as 65%.

97 MATHEMATICS AND COMPUTING↗

ECP Milestone Memo WBS 2.3.4.13 ECP/VTK-m FY18Q4 [MS-18/09-10] Dynamic Types / Rendering Topologies STDA05-16

The STDA05-16 milestone comprises the following 3 distinct deliverables. OpenMP VTK-m currently supports three types of devices: serial CPU, TBB, and CUDA. To run algorithms on multicore CPU-type devices (such as Xeon and Xeon Phi), TBB is required. However, there are known issues with integrating a software product using TBB with another one using OpenMP. Therefore, we will add an OpenMP device to the VTK-m software. When engaged, this device will run parallel algorithms using OpenMP directives. This will mesh more nicely with other code also using OpenMP. Rendering Topological Entities VTK-m currently supports surface rendering by tessellation of data structures,and rendering the resulting triangles. We will extend current functionality to include face, edge, and point rendering. Better Dynamic Types Impl For the best efficiency across all platforms, VTK-m algorithms use static typing with C++ templates. However, many libraries like VTK, ParaView, and Visit use dynamic types with virtual functions because data types often cannot be determined at compile time. We have an interface in VTK-m to merge these two typing mechanisms by generating all possible combinations of static types when faced with a dynamic type. Although this mechanism works, it generates very large executables and takes a very long time to compile. As we move forward, it is clear that these problems will get worse and become infeasible at exascale. We will rectify the problem by introducing some level of virtual methods, which require only a single code path, within VTK-m algorithms. This first milestone produces a design document to propose an approach to the new system.

97 MATHEMATICS AND COMPUTING↗

FY2011 Status Report on SN2ND Neutronics Solver Development

A change in focus of NEAMS to a broader set of reactor types necessitated significant effort being diverted away from SN2ND development in FY2011. Nevertheless, significant gains were made with regard to building the new preconditioner and meeting the objectives of 1) parallelism in energy, 2) a spatial multigrid concept, 3) using SN vector spaces to accommodate the adjoint and time dependent functionalities needed for NEAMS work, and 4) complete manuals and documentation. While the desired goal of an exportable SN2ND solver usable within the NEAMS community for neutronics analysis was not completed, we can be confident that this year’s work puts the SN2ND solver well on its way to meeting that objective. It is important to note that SN2ND is just as capable as any other methodology for doing thermal reactor analysis, but the motivation to execute on smaller scale machines has initiated focused development on simplified treatments such as those proposed in the MOCARV tool. All of the cited coding tasks were completed but not all are fully working (i.e. debugged). The spatial multigrid technique was by far the most time consuming part of the SN2ND development process. It required multiple new data structure concepts along with identifying the necessary changes in the vectors and their associated functions. Combining the spatial multigrid as part of a GMRES preconditioner operating on the full vector space, necessary for parallelization in energy, increased the complexity of that development. Research into spatial multigrid techniques for unstructured meshes, void treatments, and code optimization and acceleration are key development needs for SN2ND. Similar to previous years, a considerable amount of effort was spent on validation work. Again we focused on fast reactor problems such as ZPR experiments and the MONJU reactor in Japan. Conventional drawer homogenized models and partially heterogeneous models were used on the ZPR calculations. The drawer homogenized models produced excellent results for both eigenvalue and foil measurements on four loadings of ZPR-6/7. The partially heterogeneous models of ZPR-6/7 also produced good results, but additional work is necessary to generate consistent effective cross section data. Calculations for MONJU were homogeneous and also gave excellent results. Finally, mock-up heterogeneous calculations were setup and attempted to scope out the performance limitations of SN2ND on whole core heterogeneous calculations. Specifically, heterogeneous models of a PWR, a VHTR, and the MONJU reactor were created using anywhere from 2 group to 33 group cross section data. Meshes with greater than 100,000,000 vertices are necessary for accuracy and a specific non-scaling memory issue in PETSc restricted the SN2ND modeling (VHTR and PWR could not be solved). All of the calculations performed this year indicate that significant computational resources beyond those available today are required to obtain a high fidelity solution possible with SN2ND. The future goals for SN2ND are to get the new version fully working, reliable in execution, and to have comparable performance to the old solver such that it can be released within the NEAMS community. After its export to RSICC, we will rely upon further preconditioner research to be done in collaboration with university partners and through the exascale hub activities. Some of that research will focus on making SN2ND applicable to thermal reactor problems, spatial multigrid, and void treatments.

Smith, M. A.↗

Phase-space finite elements in a least-squares solution of the transport equation

The linear Boltzmann transport equation is solved using a least-squares finite element approximation in the space, angular and energy phase-space variables. The method is applied to both neutral particle transport and also to charged particle transport in the presence of an electric field, where the angular and energy derivative terms are handled with the energy/angular finite elements approximation, in a manner analogous to the way the spatial streaming term is handled. For multi-dimensional problems, a novel approach is used for the angular finite elements: mapping the surface of a unit sphere to a two-dimensional planar region and using a meshing tool to generate a mesh. In this manner, much of the spatial finite-elements machinery can be easily adapted to handle the angular variable. The energy variable and the angular variable for one-dimensional problems make use of edge/beam elements, also building upon the spatial finite elements capabilities. The methods described here can make use of either continuous or discontinuous finite elements in space, angle and/or energy, with the use of continuous finite elements resulting in a smaller problem size and the use of discontinuous finite elements resulting in more accurate solutions for certain types of problems. The work described in this paper makes use of continuous finite elements, so that the resulting linear system is symmetric positive definite and can be solved with a highly efficient parallel preconditioned conjugate gradients algorithm. The phase-space finite elements capability has been built into the Sceptre code and applied to several test problems, including a simple one-dimensional problem with an analytic solution available, a two-dimensional problem with an isolated source term, showing how the method essentially eliminates ray effects encountered with discrete ordinates, and a simple one-dimensional charged-particle transport problem in the presence of an electric field. (authors)

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A Testbed of Parallel Kernels for Computer Science Research

For several decades, computer scientists have sought guidance on how to evolve architectures, languages, and programming models for optimal performance, efficiency, and productivity. Unfortunately, this guidance is most often taken from the existing software/hardware ecosystem. Architects attempt to provide micro-architectural solutions to improve performance on fixed binaries. Researchers tweak compilers to improve code generation for existing architectures and implementations, and they may invent new programming models for fixed processor and memory architectures and computational algorithms. In today's rapidly evolving world of on-chip parallelism, these isolated and iterative improvements to performance may miss superior solutions in the same way gradient descent optimization techniques may get stuck in local minima. In an initial study, we have developed an alternate approach that, rather than starting with an existing hardware/software solution laced with hidden assumptions, defines the computational problems of interest and invites architects, researchers and programmers to implement novel hardware/ software co-designed solutions. Our work builds on the previous ideas of computational dwarfs, motifs, and parallel patterns by selecting a representative set of essential problems for which we provide: An algorithmic description; scalable problem definition; illustrative reference implementations; verification schemes. For simplicity, we focus initially on the computational problems of interest to the scientific computing community but proclaim the methodology (and perhaps a subset of the problems) as applicable to other communities. We intend to broaden the coverage of this problem space through stronger community involvement. Previous work has established a broad categorization of numerical methods of interest to the scientific computing, in the spirit of the NAS Benchmarks, which pioneered the basic idea of a 'pencil and paper benchmark' in the 1990s. The initial result of the more modern study was the seven dwarfs, which was subsequently extended to 13 motifs. These motifs have already been useful in defining classes of applications for architecture-software studies. However, these broad-brush problem statements often miss the nuance seen in individual kernels. For example, the computational requirements of particle methods vary greatly between the naive (but more accurate) direct calculations and the particle-mesh and particle-tree codes. Thus we commenced our study with an enumeration of problems, but then proceeded by providing not only reference implementations for each problem, but more importantly a mathematical definition that allows one to escape iterative approaches to software/hardware optimization. To ensure long term value, we have augmented each of our reference implementations with both a scalable problem generator and a verification scheme. In a paper we have prepared that documents our efforts, we describe in detail this process of problem definition, scalable input creation, verification, and implementation of reference codes for the scientific computing domain. Table 1 enumerates and describes the level of support we've developed for each kernel. We group these important kernels using the Berkeley dwarfs/motifs taxonomy using a red box in the appropriate column. As kernels become progressively complex, they build upon other, simpler computational methods. We note this dependency via orange boxes. After enumeration of the important numerical problems, we created a domain-appropriate high-level definition of each problem. To ensure future endeavors are not tainted by existing implementations, we specified the problem definition to be independent of both computer architecture and existing programming languages, models, and data types. Then, to provide context as to how such kernels productively map to existing architectures, languages and programming models, we produced reference implementations for most of the kernel (see table). These sample codes should be viewed as 'hints,' designed to show how other designers have mapped a problem's operands and operators to existing hardware and software. Since we wanted such implementations to be illustrative, we tried to ensure they were the most straightforward implementation in the easiest to understand languages using familiar architectures. To that end, most of the linear algebra-oriented computations are written in MATLAB using array indexing to process matrices, rather than one-line library calls to compute the same kernel. This ensures that the kernel's computation is explicit and readable in the implementation and not hidden behind a library. For other problems, such as the Barnes-Hut n-Body solver, the implementations were written in pure C, without any supporting library computations. Along this line, we have also created a scalable problem generator to accompany each computation.

97 MATHEMATICS AND COMPUTING↗

EMMA: an adaptive mesh refinement cosmological simulation code with radiative transfer

EMMA is a cosmological simulation code aimed at investigating the reionization epoch. It handles simultaneously collisionless and gas dynamics, as well as radiative transfer physics using a moment-based description with the M1 approximation. Field quantities are stored and computed on an adaptive three-dimensional mesh and the spatial resolution can be dynamically modified based on physically motivated criteria. Physical processes can be coupled at all spatial and temporal scales. We also introduce a new and optional approximation to handle radiation: the light is transported at the resolution of the non-refined grid and only once the dynamics has been fully updated, whereas thermo-chemical processes are still tracked on the refined elements. Such an approximation reduces the overheads induced by the treatment of radiation physics. A suite of standard tests are presented and passed by EMMA, providing a validation for its future use in studies of the reionization epoch. The code is parallel and is able to use graphics processing units (GPUs) to accelerate hydrodynamics and radiative transfer calculations. Depending on the optimizations and the compilers used to generate the CPU reference, global GPU acceleration factors between ×3.9 and ×16.9 can be obtained. As a result, vectorization and transfer operations currently prevent better GPU performance and we expect that future optimizations and hardware evolution will lead to greater accelerations.

79 ASTRONOMY AND ASTROPHYSICS↗

Integration of Heat Transfer, Stress, and Particle Trajectory Simulation

Calabazas Creek Research, Inc. developed and currently markets Beam Optics Analyzer (BOA) in the United States and abroad. BOA is a 3D, charged particle optics code that solves the electric and magnetic fields with and without the presence of particles. It includes automatic and adaptive meshing to resolve spatial scales ranging from a few millimeters to meters. It is fully integrated with CAD packages, such as SolidWorks, allowing seamless geometry updates. The code includes iterative procedures for optimization, including a fully functional, graphical user interface. Recently, time dependent, particle in cell capability was added, pushing particles synchronically under quasistatic electromagnetic fields to obtain particle bunching under RF conditions. A heat transfer solver was added during this Phase I program. Completed tasks include: (1) Added a 3D finite element heat transfer solver with adaptivity; (2) Determined the accuracy of the linear heat transfer field solver to provide the basis for development of higher order solvers in Phase II; (3) Provided more accurate and smoother power density fields; and (4) Defined the geometry using the same CAD model, while maintaining different meshes, and interfacing the power density field between the particle simulator and heat transfer solvers. These objectives were achieved using modern programming techniques and algorithms. All programming was in C++ and parallelization in OpenMP, utilizing state-of-the-art multi-core technology. Both x86 and x64 versions are supported. The GUI design and implementation used Microsoft Foundation Class.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Conforming quadrilaterals meshes on the cubed sphere.

The cubed sphere geometry, obtained by inscribing a cube in a sphere and mapping points between the two surfaces using a gnomonic (central) projection, is commonly used in atmospheric models because it is free of polar singularities and is well-suited for parallel computing. Global meshes on the cubed-sphere typically project uniform (square) grids from each face of the cube onto the sphere, and if refinement is desired then it is done with non-conforming meshes - overlaying the area of interest with a finer uniform mesh, which introduces so-called hanging nodes on edges along the boundary of the fine resolution area. An alternate technique is to tile each face of the cube with quadrilaterals without requiring the quads to be rectangular. These meshes allow for refinement in areas of interest with a conforming mesh, providing a smoother transition between high and low resolution portions of the grid than non-conforming refinement. The conforming meshes are demonstrated in HOMME, NCAR's High Order Method Modeling Environment, where two modifications have been made: the dependence on uniform meshes has been removed, and the ability to read arbitrary quadrilateral meshes from a previously-generated file has been added. Numerical results come from a conservative spectral element method modeling a selection of the standard shallow water test cases.

99 GENERAL AND MISCELLANEOUS↗

ATDM LANL FleCSI: Topology and Execution Framework

FleCSI is a compile-time configurable C++ framework designed to support multi-physics application development. As such, FleCSI attempts to provide a very general set of infrastructure design patterns that can be specialized and extended to suit the needs of a broad variety of solver and data requirements. This means that FleCSI is potentially useful to many different ECP projects. Current support includes multidimensional mesh topology, mesh geometry, and mesh adjacency information, n-dimensional hashed-tree data structures, graph partitioning interfaces, and dependency closures (to identify data dependencies between distributed-memory address spaces). FleCSI introduces a functional programming model with control, execution, and data abstractions that are consistent with state-of-the-art task-based runtimes such as Legion and Charm++. The model also provides support for fine-grained, data-parallel execution with backend support for runtimes such as OpenMP and C++17. The FleCSI abstraction layer provides the developer with insulation from the underlying runtimes, while allowing support for multiple runtime systems, including conventional models like asynchronous MPI. The intent is to give developers a concrete set of user-friendly programming tools that can be used now, while allowing flexibility in choosing runtime implementations and optimizations that can be applied to architectures and runtimes that arise in the future. This project is essential to the ECP Ristra Next-Generation Code project, part of ASC ATDM, because it provides a hierarchically parallel programming model that is consistent with the design of modern system architectures, but which allows for the straightforward expression of algorithmic parallelism in a portably performant manner.

97 MATHEMATICS AND COMPUTING↗

Evaluation of SNS Beamline Shielding Configurations using MCNPX Accelerated by ADVANTG

Shielding analyses for the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory pose significant computational challenges, including highly anisotropic high-energy sources, a combination of deep penetration shielding and an unshielded beamline, and a desire to obtain well-converged nearly global solutions for mapping of predicted radiation fields. The majority of these analyses have been performed using MCNPX with manually generated variance reduction parameters (source biasing and cell-based splitting and Russian roulette) that were largely based on the analyst's insight into the problem specifics. Development of the variance reduction parameters required extensive analyst time, and was often tailored to specific portions of the model phase space. We previously applied a developmental version of the ADVANTG code to an SNS beamline study to perform a hybrid deterministic/Monte Carlo analysis and showed that we could obtain nearly global Monte Carlo solutions with essentially uniform relative errors for mesh tallies that cover extensive portions of the model with typical voxel spacing of a few centimeters. The use of weight window maps and consistent biased sources produced using the FW-CADIS methodology in ADVANTG allowed us to obtain these solutions using substantially less computer time than the previous cell-based splitting approach. While those results were promising, the process of using the developmental version of ADVANTG was somewhat laborious, requiring user-developed Python scripts to drive much of the analysis sequence. In addition, limitations imposed by the size of weight-window files in MCNPX necessitated the use of relatively coarse spatial and energy discretization for the deterministic Denovo calculations that we used to generate the variance reduction parameters. We recently applied the production version of ADVANTG to this beamline analysis, which substantially streamlined the analysis process. We also tested importance function collapsing (in space and energy) capabilities in ADVANTG. These changes, along with the support for parallel Denovo calculations using the current version of ADVANTG, give us the capability to improve the fidelity of the deterministic portion of the hybrid analysis sequence, obtain improved weight-window maps, and reduce both the analyst and computational time required for the analysis process.

43 PARTICLE ACCELERATORS↗

In situ multi-tier auto-ignition detection applied to dual-fuel combustion simulations

Here we use an anomaly detection methodology that is centered on analyzing fourth-order joint moments (co-kurtosis), particularly focusing on its application in auto-ignition of combustion problems with large numbers of species. Unsupervised anomaly detection is challenging to generalize across problem types and domains. A recent technique, centered on analyzing information in the fourth-order joint moment co-kurtosis, has shown promise, especially for high-dimensional scientific data. In this work we present developments to the co-kurtosis based anomaly detection method needed to make it effective and scalable for large-scale distributed scientific data, such as those generated by massively parallel simulations. An in situ co-kurtosis algorithm is employed as the anomaly detection method for identifying ignition kernels in simulations of turbulent combustion. Here, we extend an existing methodology which identifies regions of the domain where anomalies are present, and add another tier of anomaly detection where the individual samples contributing to the anomaly are identified. We apply this algorithm on-the-fly to a variety of turbulent reacting flow problems and compare it to the widely used (but significantly more expensive) chemical explosive mode analysis (CEMA). We demonstrate the ability of the method to detect and identify the onset of low and high temperature ignition which can be used for computational steering, as chemical and combustion anomalies occur intermittently at spatio-temporal locations unknown a priori. Finally, we apply our lightweight in situ algorithm to an exascale high-fidelity simulation with a total of 2.4 Trillion degrees of freedom, performed using an adaptive mesh refinement solver. Furthermore, through a scalability analysis, we show that the relative computational cost of this in-situ anomaly detection algorithm compared to an iteration of the reacting flow solver is negligible.

97 MATHEMATICS AND COMPUTING↗