Flexible Spatial Partitions in MPACT Through Module-Based Data Passing

Abstract

As part of the Consortium for Advanced Simulation of Light Water Reactors (CASL), the Virtual Environment for Reactor Applications (VERA) is being developed to provide high-fidelity multiphysics simulations of nuclear reactor cores. The MPACT code being developed collaboratively by Oak Ridge National Laboratory and the University of Michigan is the primary deterministic neutron transport solver available within VERA. MPACT employs the 2D/1D method to solve 3D problems using 2D method of characteristics (MOC) radial transport solvers with 1D nodal expansion method-based P3 (NEM-P3) axial transport solvers, which are coupled using axial and radial transverse leakages. A 3D coarse mesh finite difference (CMFD) solver is used to provide accelerated convergence and stability. Quarter-core problems in MPACT are typically run on moderately sized clusters ({approx}500-10,000 cores). As such, it relies heavily on spatial decomposition, both radially and axially, using MPI data communication along spatial domain boundaries to pass angular flux boundary condition and neighboring data. The initial spatial decomposition strategy implemented into MPACT overlaid a Cartesian grid over the core to form the domain boundaries. With this approach, a single buffer spanning across the entire boundary between two spatial domains is passed to a domain's neighbor. Additionally, only one neighboring domain index was allowed in each direction. For example, the entire eastern boundary of a domain must be the entire western boundary of the corresponding neighbour's domain. This implementation helped to execute cases on several thousand cores, where substantial radial decomposition was used. The default for several years was to use assembly decomposition, which yielded 73 radial partitions in the Watts Bar Unit 1 (WBN1) models being analyzed. However, as CASL formed stronger connections with industry partners, there was a strong push to allow MPACT to run on fewer cores, preferably in the 500-1,000 core range. Such cases require much less radial decomposition, typically with 8-16 radial domains on each plane. Unfortunately, the previous decomposition strategy is very limited in its ability to yield well-balanced domains, and it does not permit partitions with an arbitrary number of domains, particularly with small numbers of radial domains (3, 5, 7, 11, 13, etc.). With these limitations, a more flexible approach that provided more even-load balancing was needed. The details of this new strategy are outlined here, along with insights to its advantages. The examples provided are for the WBN1 core, which has been the focus of several initial CASL analyses. It is worth noting that spatial decomposition strategies have been employed in many codes with focus on a number of different transport methodologies. The references cited are not exhaustive as this has been very important research topic across many areas.