Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel mesh generation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Documentation for MeshKit - Reactor Geometry (&mesh) Generator

This report gives documentation for using MeshKit’s Reactor Geometry (and mesh) Generator (RGG) GUI and also briefly documents other algorithms and tools available in MeshKit. RGG is a program designed to aid in modeling and meshing of complex/large hexagonal and rectilinear reactor cores. RGG uses Argonne’s SIGMA interfaces, Qt and VTK to produce an intuitive user interface. By integrating a 3D view of the reactor with the meshing tools and combining them into one user interface, RGG streamlines the task of preparing a simulation mesh and enables real-time feedback that reduces accidental scripting mistakes that could waste hours of meshing. RGG interfaces with MeshKit tools to consolidate the meshing process, meaning that going from model to mesh is as easy as a button click. This report is designed to explain RGG v 2.0 interface and provide users with the knowledge and skills to pilot RGG successfully. Brief documentation of MeshKit source code, tools and other algorithms available are also presented for developers to extend and add new algorithms to MeshKit. RGG tools work in serial and parallel and have been used to model complex reactor core models consisting of conical pins, load pads, several thousands of axially varying material properties of instrumentation pins and other interstices meshes.

Jain, Rajeev↗

Developing And Scaling an OpenFOAM Model to Study Turbulent Flow in a HFIR Coolant Channel

Improving the understanding of how computational fluid dynamics (CFD) direct numerical simulations (DNS) of flows in the High Flux Isotope Reactor (HFIR) perform when run in parallel using the high performance computing (HPC) platform Summit at the Oak Ridge Leadership Computing Facility (OLCF) is of particular importance to boost the computational tools used to support HFIR conversion to low enriched fuel (LEU). Evaluation of scaling performance was driven by the increasing importance of graphics processing unit (GPU) usage in HPC, which is becoming the standard for modern supercomputers such as Summit. The desired results are to obtain a strong positive correlation between the computational resources dedicated to a problem and the relative speed-up of the simulation in comparison to a benchmark. This capability will allow substantially improvement in HFIR flow analytical capabilities, specifically when predicting turbulence properties at high Reynolds numbers. The study leverages previous simulation results performed with code PHASTA (finite element) on HPC platforms Cori (NERSC) and Theta (ALCF) [1] with computing options provided in the computing platform OpenFOAM (finite volume) at OLCF. Transitioning from PHASTA to OpenFOAM will (1) eliminate dependence on third-party software for mesh generation and manipulation, (2) reduce resource needs by employing modern architectures, and (3) build expertise for future modeling of HFIR-specific problems like heat transfer in involute geometry, entrance effects, flow structure in channel corners, and so on—all important issues when defining the available thermal margins in the transition to LEU. CPUs and GPUs differ significantly in their architecture and utilization, as discussed in the literature [2]. The most important differences are in the approach to computations and their memory. A single GPU contains a large quantity of cores, enabling it to perform with a much higher throughput than a CPU, but execution requires a different approach. GPU codes execute instructions using the Single-Instruction Multiple-Thread (SIMT) approach in which a single instruction is used for groups of threads called warps. A warp typically consists of 32 threads which must execute the same set of instructions, although on separate threads. Alternately, a CPU has far fewer cores that are much more flexible in their operation, excelling at quickly performing more complex serial computations. This is why GPUs have greater throughput when properly utilized. The second important difference is seen when comparing their memory spaces. Limited memory allocations and CPU–GPU communications cause a significant bottleneck in GPU-accelerated programs. Further study was required to properly take advantage of GPU resources. A comprehensive analysis of code performance and the model-specific features of turbulence constitutes the core of this work. In this study, a DNS simulation of HFIR channel turbulence was performed with the finite volume CFD code OpenFOAM v2112 and CUDA v11.0 on Red Hat Enterprise Linux v8.2. The OpenFOAM installation had AMGx integrated to enable GPU acceleration and utilizes the PETSc4FOAM library. The computational resources and the problem size were scaled on CPU and CPU + GPU architectures to gain a better understanding of the performance of a DNS problem on modern computing hardware. The study aimed to analyze the scaling of the code exclusively on CPUs and then to examine the scaling of the codes with GPU acceleration enabled. Scaling studies included CPU and GPU acceleration on a mesh of varying resolution to analyze the impact of problem size relative to computational resources. In the course of preparing the GPU configuration on Summit, mainly using the AMGX solvers, difficulties were encountered stemming from constant changes resulting from extensive ongoing development activities and the changing environment. This resulted in the inability to complete the GPU portion of the work. The code was compiled and tested, but production runs to assess acceleration were not performed because the used discretional compute time allocation expired as year-end approached. The Summit HPC platform is scheduled for decommissioning in 2024, making it unattractive for future use with Nvidia-based GPUs. Therefore, the work will be moved onto NERSC machines in FY24. An application was prepared and submitted, and sufficient node-hours were awarded to continue the research in the next calendar year. This report summarizes work performed thus far, which mostly focused on CPU OpenFOAM computing.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Modeling the Structural Response from a Propagating High Explosive Using Smooth Particle Hydrodynamics

This report primarily concerns the use of two massively parallel finite element codes originally written and maintained at Lawrence Livermore National Laboratory. ALE3D is an explicit hydrodynamics code commonly employed to simulate wave propagation from high energy scenarios and the resulting interaction with nearby structures. This coupled response ensures that a structure is accurately applied with a blast loading varying both in space and time. Figure 1 illustrates the radial outward propagation of a pressure wave due to a center detonated spherical explosive originating from the lower left. The radial symmetry seen in this scenario is lost when instead a cylindrocal charge is detonated. Figure 2 indicates that a stronger, faster traveling pressure wave occurs in the direction of the normal axis to the cylinder. The ALE3D name is derived because of the use of arbitrary-Lagrange-Eulerian elements in which the mesh is allowed to advect; a process through which the mesh is modified to alleviate tanlging and general mesh distortion often cuased by high energy scenarios. The counterpart to an advecting element is a Lagrange element, whose mesh moves with the material. Ideally all structural components are kept Lagrange as long as possible to preserve accuracy of material variables and minimize advection related errors. Advection leads to mixed zoning, so using structural Lagrange elements also improves the visualization when post processing the results. A simplified representation of the advection process is shown in Figure 3. First the mesh is distorted due to material motion during the Lagrange step. The mesh is then shifted to an idealized and less distorted state to prevent irregular zones caused by the Lagrange motion. Lastly, the state variables are remapped to the elements of the newly constructed mesh. Note that Figure 3 represents a purely Eulerian mesh relaxation because the mesh is relocated back to the pre-Lagrange position. This is the case when the material flows through a still mesh. This is not typically done in an ALE3D analysis, especially if Lagrange elements exist. Deforming Lagrange elements would certainly tangle with a Eulerian mesh eventually. The best method in this case is to have an advecting mesh positioned as some relaxed version of the pre and post Lagrange step; this gives the best opportunity of modeling a high energy event with a combination of Lagrange and ALE elements. Dyne3D is another explicit dynamic analysis code, ParaDyn being the parallel version. ParaDyn is used for predicting the transient response of three dimensional structures using Lagrangian solid mechanics. Large deformation and mesh tangling is often resolved through the use of an element deletion scheme. This is useful to accommodate component failure, but if it is done purely as a means to preserve a useful mesh it can lead to problems because it does not maintain continuity of the material bulk response. Whatever medium exists between structural components is typically not modeled in ParaDyn. Instead, a structure either has a known loading profile applied or given initial conditions. The many included contact algorithms can calculate the loading response of materials if and when they collide. A recent implementation of an SPH module in which failed or deleted material nodes are converted to independent particles is currently being utilized for a variety of spall related problems and high velocity impact scenarios. Figure 4 shows an example of a projectile, given an initial velocity, and how it fails the first plate which generates SPH particles which then interact with and damage the second plate.

36 MATERIALS SCIENCE↗

Initial Investigation of Coupled Electron-Photon Transport with SCEPTRE and RAPTURE (Rev.1.1)

The SCEPTRE1 code models coupled photon and electron transport in material media using discrete ordinates methods. The code can use different solvers in 1D and 3D in parallel, promising users the ability to run problems that are significant both in physics and computational scale. One capability that is perhaps unique at this time in the NNSA complex is SCEPTRE’s coupled photon– electron transport physics in a discrete ordinates approach. For example, incident x–rays can generate a significant electron population, whose transport can be simulated within the same code. In this context, fully–coupled means that photons can produce electrons and electrons can produce photons; uncoupled or no–coupling means photons do not produce electrons. Partial–coupling, an intermediate option in which photons can produce electrons but electrons do not produce photons, is not considered in the present work. The MCNP code can treat this physics with a Monte Carlo approach. MCNP should be viewed as complementary to SCEPTRE in both computational method (Monte Carlo versus discrete ordinates) as well as data libraries. SCEPTRE and MCNP should provide important checks on each other, especially from complementary modeling frameworks of a S n (discrete ordinates) mesh based method versus Monte Carlo particle based method. In order to learn more about SCEPTRE, and how it might inform engineering work at the Laboratory, we used it to carry out a set of initial studies. This was a series of 1D simulations of photon beams hitting Aluminum slabs. Our studies mainly used SCEPTRE directly, and made little use of the closely associated RAPTURE code (more on this below). The goals of these studies were first, to familiarize ourselves with operating the code, and second, to begin to assess the importance of coupled electron–photon transport. To support these studies, we began to develop a set of scripts to initialize inputs, run tools, process outputs, and plot results. An initial version of these scripts has been committed to the Lightning workflow repository. This memo relates the results of those studies.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Developing Uintah's Runtime System for Forthcoming Architectures

The need to solve ever-more complex science and engineering problems on computer architectures that are rapidly evolving poses considerable challenges for modern simulation frameworks and packages. While the decomposition of software into a programming model that generates a task-graph for execution by a runtime system make portability and performance possible, it also imposes considerable demands on that runtime system. Here, the challenge of meeting those demands for near-term and longer term systems in the context of the Uintah framework is considered in key areas such as support for data structures, tasks on heterogeneous architectures, performance portability, power management and designing for resilience by the use of replication based upon adaptive mesh refinement processes is addressed. Examples of techniques in these areas are used to illustrate performance improvements obtained for present large-scale systems and the needs of proposed systems considered.

97 MATHEMATICS AND COMPUTING↗

TRIPOLI-4{sup R} version 9 Shielding for the Shielding Community: Overview, Relevant Benchmarking for Shielding and Licensing Policy - Paper 49

This is a review paper of the Shielding only version of TRIPOLI-4. TRIPOLI-4 and TRIPOLI-4 S: TRIPOLI{sup R} represents a family of radiation transport codes using the Monte Carlo method (1,2). It has been continuously developed at CEA since the mid 60's. TRIPOLI-4{sup R}, being the fourth generation, has been developed starting from the mid 90's in C++. It tracks neutrons from 20 MeV down to 10{sup -5} eV, photons, electrons and positrons from 100 MeV to 1 keV, in 3D arbitrary geometries either in TRIPOLI{sup R} proprietary format or ROOT(4) format. It reads its continuous energy nuclear data from any evaluation in ENDF format, including but not limited to JEFF-3.1.1 and ENDF/BVII.0. It has various simulation modes, fixed-source (with or without fission), criticality, or both of them chained. A variety of tallies are available: averaged flux in a volume (track length and collision estimators), surface flux and current, point flux, reaction rates, deposited energy, dpa, pka, gamma spectroscopy, mesh tallies, tallies for criticality (k{sub eff}, beta eff, kinetic parameters). It provides the user with embedded response functions from IRDF and EAF as well. Several variance reduction techniques (3), including automated ones, make it particularly suitable to deep penetration problems (implicit capture, Russian roulette, exponential biasing with automatic estimation of the importance map). It may run in parallel, on single multi-core machines as well as heterogeneous networks of stations or massively parallel machines. It comes with productivity tools designed to speed-up the elaboration and verification of the input deck as well as the output analysis: 2D interactive display of geometry; 2D display of mesh tallies and collision sites; 3D input deck modeler; compatibility with ROOT tools to post-process TRIPOLI{sup R} outputs. Its V and V test base is made of more than 1000 internationally accepted benchmarks from ICSBEP and SINBAD (REPLICA, ASPIS, NESDIP) and covers all its application domains, including several shielding cases which will be detailed in the talk. TRIPOLI-4{sup R} is the reference Monte Carlo code for CEA, EDF, and also for AREVA for a number of applications. A 'Shielding-only' version has been designed to specifically answer the needs of the shielding community. It is distributed through the NEA Databank with a license for R and D, covering the fusion applications, whose terms are detailed in the paper.

61 RADIATION PROTECTION AND DOSIMETRY↗

MOOSE framework enhancements for meshing reactor geometries

MOOSE is an open-source, parallel finite element framework designed to permit rapid development of robust multi-physics modeling capabilities. Under the DOE-NEAMS program, numerous solvers have been developed utilizing the open-source MOOSE framework for multiphysics reactor analysis. These solvers require input finite element meshes representing the discretized geometry. Typically, reactor analysts turn to licensed external tools for creation of reactor geometry meshes. Recently, enhancements have been added to the MOOSE framework to mesh common reactor geometries and improve MOOSE-based application user workflows. Support for hexagonal pins, assemblies, and cores has been added, and Cartesian support has been extended. Options for modeling static and rotating control drums within a hexagonal assembly are now available. Pin, assembly, and plane regions can be identified through automatically applied tags on the mesh called 'reporting IDs' for easier post- processing of physics results. An external open-source triangle routine has been leveraged within MOOSE to mesh core periphery zones. A set of reactor geometry builder routines further streamline the construction of hexagonal and Cartesian cores and include the ability to assign materials to regions during mesh generation. The new meshing routines are available through the MOOSE framework in the open-source 'Reactor' module, and the resulting directly within MOOSE-based applications or exported as Exodus II files for use in other finite element solvers. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Reference guide to WPP version 2.0

WPP is a computer program for simulating seismic wave propagation on parallel machines. WPP solves the governing equations in second order formulation using a node-based finite difference approach. The basic numerical method is described in [9]. WPP implements substantial capabilities for 3-D seismic modeling, with a free surface condition on the top boundary, non-reflecting far-field boundary conditions on the other boundaries, point force and point moment tensor source terms with many predefined time dependencies, fully 3-D heterogeneous material model specification, output of synthetic seismograms in the SAC [4] format, output of GMT [11] scripts for laying out simulation information on a map, and output of 2-D slices of (derived quantites of) the solution field as well as the material model. Version 2.0 of WPP allows the free surface boundary condition to be imposed on a curved topography. For this purpose a curvilinear mesh is used near the free surface, extending into the computational domain down to a user specified level. The elastic wave equations and the free surface boundary conditions are discretized on the curvilinear mesh using the energy conserving technique described in [2]. A curvilinear mesh generator is built into WPP and the curvilinear mesh is automatically generated from the topography. Below the curvilinear grid, the elastic wave equation is discretized on Cartesian meshes, which leads to a more computationally efficient algorithm. In version 2.0 of WPP, Cartesian local mesh refinement can be used to make the computational mesh finer near the free surface, where more resolution often is needed to resolve short wave lenghts in the solution, for example in sedimentary basins. The mesh refinement is performed in the vertical direction and each Cartesian grid is constructed from user specified refinement levels. In this approach, the grid size in all three spatial directions is doubled across each mesh refinement interface, leading to substantial savings in memory and computational effort. The energy conserving mesh refinement coupling method described in [10] is used to handle the hanging nodes along the refinement interface. The examples subdirectory of the WPP source distribution contains several examples and validation tests. Many Matlab/octave scripts are provided in the tools directory.

58 GEOSCIENCES↗

Analyzing and visualizing next generation climate data (Final Report)

The project "Analyzing and visualizing next generation climate data" adds block-structured (mosaic) grid support, parallel processing, and 2D/3D curvilinear interpolation to the open-source UV-CDAT climate data analysis tool. Block structured grid support complies to the Gridspec extension submitted to the Climate and Forecast metadata conventions. It contains two parts: aggregation of data spread over multiple mosaic tiles (M-SPEC) and aggregation of temporal data stored in different files (F-SPEC). Together, M-SPEC and F-SPEC allow users to interact with data stored in multiple files as if the data were in a single file. For computational expensive tasks, a flexible, multi-dimensional, multi-type distributed array class allows users to process data in parallel using remote memory access. Both nodal and cell based interpolation is supported; users can choose between different interpolation libraries including ESMF and LibCF depending on the their particular needs.

54 ENVIRONMENTAL SCIENCES↗

Efficient exascale discretizations: High-order finite element methods

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.

97 MATHEMATICS AND COMPUTING↗

Mapping of orthogonal 2D flux coordinates for two nearby magnetic X-points to logically rectangular domains

The behavior of plasmas in configurations with magnetic null points is of potential interest for a number of systems. Such nulls, sometimes referred to as X-points, a name strictly valid only for wellseparated nulls where magnetic separatrices cross at a right angle, may occur in only one component of the magnetic field, such as the poloidal field in a tokamak. Numerical simulations are often used to study and predict turbulence and transport in such systems. For cases with strong magnetic field such that the ion and electron gyro-frequencies are much larger that particle collision frequencies, there is a strong anisotropy in the plasma transport along the magnetic field compared to across the field. Because of the strong anisotropy, it is often advantageous for simulations to use a coordinate system based on magnetic flux surfaces together with a coordinate orthogonal to the flux surfaces. This report presents the topological considerations needed to construct the mapping on an orthogonal flux-surface mesh onto a logically rectangular domain often used in simulation codes of the divertor region of tokamaks having two nearby X-points, thus extending the mapping now common single-null or well-separated double nullpoint cases with one null located at the top and bottom of the device. This extension is especially relevant to study the capability of snowflake divertors in tokamaks for reducing the peak heat flux of exhaust plasma on the target plates [1-3]. The analysis presented here follows the analytic description of the poloidal magnetic field, B pol , near two closely-spaced X-points given by Ryutov [1-3]. This analysis is applicable to experimental devices where the two X-points are generated by distributed poloidal field coils and the plasma core current. This analytic description is used to identify topological features of an orthogonal mesh. A key parameter that specifies the various topology changes is the poloidal angle between the primary X-point and the second X-point, and the changes are associated with a set of critical angles. Understanding these changes leads to the final step in the mesh construction by identifying the locations of various regions or patches in configuration space and in a corresponding logically rectangular index space. UEDGE [4] and other 2-D fluid plasma simulation codes [5-7] describe the edge plasma of toroidally symmetric tokamak configurations with one or more X-points in a divertor region. The 2-D spatial mesh is usually based on a magnetic configuration produced by an MHD equilibrium code such as EFIT and written out as one or more “eqdsk” files. As illustrated in Fig. 1, the domain of the simulated plasma consists of several flux tubes bounded by poloidal magnetic flux surfaces. The number and spacing of the flux surfaces defines the “radial” resolution of the mesh. The “radial” direction is locally orthogonal to flux surfaces; the “poloidal” direction is locally parallel to flux surfaces. The “poloidal” resolution of the mesh is defined by the number of cells in the poloidal direction between adjacent flux surfaces. For boundary plasma transport codes used for tokamak configurations such as UEDGE it is convenient, though not necessary, to assume that all flux tubes have the same number of poloidal cells so the mesh is logically rectangular in index space, but does possess internal boundaries linking different sub-rectangles in the computational domain. The following sections of this report describe indexing schemes that map tokamak magnetic configurations with one or more x-points in the divertor region(s) onto a rectangular index space. Simulation results from the UEDGE code illustrate the schemes for a few configurations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

TH-E-18A-01: Developments in Monte Carlo Methods for Medical Imaging

Monte Carlo simulation methods are widely used in medical physics research and are starting to be implemented in clinical applications such as radiation therapy planning systems. Monte Carlo simulations offer the capability to accurately estimate quantities of interest that are challenging to measure experimentally while taking into account the realistic anatomy of an individual patient. Traditionally, practical application of Monte Carlo simulation codes in diagnostic imaging was limited by the need for large computational resources or long execution times. However, recent advancements in high-performance computing hardware, combined with a new generation of Monte Carlo simulation algorithms and novel postprocessing methods, are allowing for the computation of relevant imaging parameters of interest such as patient organ doses and scatter-to-primaryratios in radiographic projections in just a few seconds using affordable computational resources. Programmable Graphics Processing Units (GPUs), for example, provide a convenient, affordable platform for parallelized Monte Carlo executions that yield simulation times on the order of 10{sup 7} xray/ s. Even with GPU acceleration, however, Monte Carlo simulation times can be prohibitive for routine clinical practice. To reduce simulation times further, variance reduction techniques can be used to alter the probabilistic models underlying the x-ray tracking process, resulting in lower variance in the results without biasing the estimates. Other complementary strategies for further reductions in computation time are denoising of the Monte Carlo estimates and estimating (scoring) the quantity of interest at a sparse set of sampling locations (e.g. at a small number of detector pixels in a scatter simulation) followed by interpolation. Beyond reduction of the computational resources required for performing Monte Carlo simulations in medical imaging, the use of accurate representations of patient anatomy is crucial to the virtual generation of medical images and accurate estimation of radiation dose and other imaging parameters. For this, detailed computational phantoms of the patient anatomy must be utilized and implemented within the radiation transport code. Computational phantoms presently come in one of three format types, and in one of four morphometric categories. Format types include stylized (mathematical equation-based), voxel (segmented CT/MR images), and hybrid (NURBS and polygon mesh surfaces). Morphometric categories include reference (small library of phantoms by age at 50th height/weight percentile), patient-dependent (larger library of phantoms at various combinations of height/weight percentiles), patient-sculpted (phantoms altered to match the patient's unique outer body contour), and finally, patient-specific (an exact representation of the patient with respect to both body contour and internal anatomy). The existence and availability of these phantoms represents a very important advance for the simulation of realistic medical imaging applications using Monte Carlo methods. New Monte Carlo simulation codes need to be thoroughly validated before they can be used to perform novel research. Ideally, the validation process would involve comparison of results with those of an experimental measurement, but accurate replication of experimental conditions can be very challenging. It is very common to validate new Monte Carlo simulations by replicating previously published simulation results of similar experiments. This process, however, is commonly problematic due to the lack of sufficient information in the published reports of previous work so as to be able to replicate the simulation in detail. To aid in this process, the AAPM Task Group 195 prepared a report in which six different imaging research experiments commonly performed using Monte Carlo simulations are described and their results provided. The simulation conditions of all six cases are provided in full detail, with all necessary data on material composition, source, geometry, scoring and other parameters provided. The results of these simulations when performed with the four most common publicly available Monte Carlo packages are also provided in tabular form. The Task Group 195 Report will be useful for researchers needing to validate their Monte Carlo work, and for trainees needing to learn Monte Carlo simulation methods. In this symposium we will review the recent advancements in highperformance computing hardware enabling the reduction in computational resources needed for Monte Carlo simulations in medical imaging. We will review variance reduction techniques commonly applied in Monte Carlo simulations of medical imaging systems and present implementation strategies for efficient combination of these techniques with GPU acceleration. Trade-offs involved in Monte Carlo acceleration by means of denoising and “sparse sampling” will be discussed. A method for rapid scatter correction in cone-beam CT (<5 min/scan) will be presented as an illustration of the simulation speeds achievable with optimized Monte Carlo simulations. We will also discuss the development, availability, and capability of the various combinations of computational phantoms for Monte Carlo simulation of medical imaging systems. Finally, we will review some examples of experimental validation of Monte Carlo simulations and will present the AAPM Task Group 195 Report. Learning Objectives: Describe the advances in hardware available for performing Monte Carlo simulations in high performance computing environments. Explain variance reduction, denoising and sparse sampling techniques available for reduction of computational time needed for Monte Carlo simulations of medical imaging. List and compare the computational anthropomorphic phantoms currently available for more accurate assessment of medical imaging parameters in Monte Carlo simulations. Describe experimental methods used for validation of Monte Carlo simulations in medical imaging. Describe the AAPM Task Group 195 Report and its use for validation and teaching of Monte Carlo simulations in medical imaging.

60 APPLIED LIFE SCIENCES↗

Efficacy of the Cell Perturbation Method in Large-Eddy Simulations of Boundary Layer Flow over Complex Terrain

A challenge to simulating turbulent flow in multiscale atmospheric applications is the efficient generation of resolved turbulence motions over an area of interest. One approach is to apply small perturbations to flow variables near the inflow planes of turbulence-resolving simulation domains nested within larger mesoscale domains. While this approach has been examined in numerous idealized and simple terrain cases, its efficacy in complex terrain environments has not yet been fully explored. Here, we examine the benefits of the stochastic cell perturbation method (CPM) over real complex terrain using data from the 2017 Perdigão field campaign, conducted in an approximately 2-km wide valley situated between two nearly parallel ridges. Following a typical configuration for multiscale simulation using nested domains within the Weather Research and Forecasting (WRF) model to downscale from the mesoscale to a large-eddy simulation (LES), we apply the CPM on a domain with horizontal grid spacing of 150 m. At this resolution, spurious coherent structures are often observed under unstable atmospheric conditions with moderate mean wind speeds. Results from such an intermediate resolution grid are often nested down for finer, more detailed LES, where these spurious structures adversely affect the development of turbulence on the subsequent finer grid nest. We therefore examine the impacts of the CPM on the representation of turbulence within the nested LES domain under moderate mean flow conditions in three different stability regimes: weakly convective, strongly convective, and weakly stable. In addition, two different resolutions of the underlying terrain are used to explore the role of the complex topography itself in generating turbulent structures. We demonstrate that the CPM improves the representation of turbulence within the LES domain, relative to the use of high-resolution complex terrain alone. During the convective conditions, the CPM improves the rate at which smaller-scales of turbulence form, while also accelerating the attenuation of the spurious numerically generated roll structures near the inflow boundary. During stable conditions, the coarse mesh spacing of the intermediate LES domain used herein was insufficient to maintain resolved turbulence using CPM as the flow develops downstream, highlighting the need for yet higher resolution under even weakly stable conditions, and the importance of accurate representation of flow on intermediate LES grids.

17 WIND ENERGY↗

Langmuir turbulence and filament frontogenesis in the oceanic surface boundary layer

Submesoscale currents, small-scale turbulence and surface gravity waves co-exist in the upper ocean and interact in complex ways. To expose the couplings, the frontogenetic life cycle of an idealized cold dense submesoscale filament interacting with upper ocean Langmuir turbulence is investigated in large-eddy simulations (LESs) based on the incompressible wave-averaged equations. The simulations utilize large domains and fine meshes with $6.4\times 10^{9}$ grid points. Case studies are made with surface winds or surface cooling with waves oriented in across-filament (perpendicular) or down-filament (parallel) directions relative to the two-dimensional filament axis. The currents $u$ , $v$ and $w$ are aligned with the across-filament, down-filament and vertical directions, respectively. Frontogenesis is induced by across-filament Lagrangian secondary circulations in the boundary layer, and it is shown to be strongly impacted by surface waves, in particular the propagation direction relative to the filament axis. In a horizontally heterogeneous boundary layer, surface waves induce both mean and fluctuating Stokes-drift vortex forces that modify a linear, hydrostatic turbulent thermal wind (TTW) approximation for momentum. Down-filament winds and waves are found to be especially impactful, they significantly reduce the peak level of frontogenesis by fragmenting the filament into primary and secondary down-welling sites in a broad frontal zone over a width ${\sim}500~\text{m}$ . At peak frontogenesis, opposing down-filament jets $\langle v\rangle$ overlie each other resulting in a vigorous vertical shear layer $\unicode[STIX]{x2202}_{z}\langle v\rangle$ with large vertical momentum flux $\langle v^{\prime }w^{\prime }\rangle$ . Filament arrest is induced by a lateral shear instability that generates horizontal momentum flux $\langle u^{\prime }v^{\prime }\rangle$ at low wavenumbers. The turbulent vertical velocity patterns, indicative of coherent Langmuir cells, change markedly across the horizontal domain with both across-filament and down-filament winds under the action of submesoscale currents.

Mechanics↗

A Scalable Algorithm for Radiative Heat Transfer Using Reverse Monte Carlo Ray Tracing

Radiative heat transfer is an important mechanism in a class of challenging engineering and research problems. A direct all-to-all treatment of these problems is prohibitively expensive on large core counts due to pervasive all- to-all MPI communication. The massive heat transfer problem arising from the next generation of clean coal boilers being modeled by the Uintah framework has radiation as a dominant heat transfer mode. Reverse Monte Carlo ray tracing (RMCRT) can be used to solve for the radiative-flux divergence while accounting for the effects of participating media. The ray tracing approach used here replicates the geometry of the boiler on a multi-core node and then uses an all-to-all communication phase to distribute the results globally. The cost of this all-to-all is reduced by using an adaptive mesh approach in which a fine mesh is only used locally, and a coarse mesh is used elsewhere. Here, a model for communication and computation complexity is used to predict performance of this new method. We show this model is consistent with observed results and demonstrate excellent strong scaling to 262K cores on the DOE Titan system on problem sizes that were previously computationally intractable.

97 MATHEMATICS AND COMPUTING↗

Development Status of PROTEUS-MOC

The PROTEUS-MOC solver is intended to resolve the outstanding computational challenges that the PROTEUS-SN is limited by and bridge the gap between PROTEUS-SN and existing homogenized structured grid solvers. The primary push behind all of this work is to carry out coupled thermal-mechanical, fluid dynamics, and neutronics physics problems. The PROTEUS-MOC development has taken advantage of the combined experience with PROTEUS-SN and DeCART with the goal of producing a tool which combines the best features of both tools. Taking the DeCART tool as the model, it merges a subgroup cross section interpolation technique with a 2D method of characteristics approach and a 1D SN formulation. DeCART has been specifically demonstrated to be accurate on PWR and VHTR problems, but it has displayed several issues with regard to accuracy on thin axial mesh sizes which are attributable to the 2D1D scheme it is built upon. There are also notable problems with performance and scalability all of which we hope to resolve in PROTEUS-MOC. PROTEUS-MOC is at its core, based upon the success achieved with MOCFE which demonstrated excellent parallelization with respect to angle and good parallelization with respect to space. More recent efforts have indicated parallelization with respect to energy is also feasible. The work completed yielded a novel methodology where the axial basis functions are treated separately from the radial ones, but the geometry is extruded. This eliminates the need to carry out and store 3D ray tracing data and simultaneously allows the coarse axial node sizes typical with DeCART. To rapidly build a prototype, a standard power iteration algorithm was constructed using an unpreconditioned GMRES solution algorithm for solving the flux vector in each fission source iteration. The new PROTEUS-MOC solver was deployed on two test problems: Takeda and a heterogeneous cylindrical heavy water system. The Takeda benchmark problem is a standard homogeneous test problem which was previously found to be too difficult for the 3D MOCFE code to solve. Using the PROTEUS-MOC we were able to show space-angle convergence although full convergence requires further improvements in performance. The second benchmark problem was constructed more for future work than this year, but the results generated are promising indicating that it will be very useful in highlighting the weaknesses and strengths of the PROTEUS-MOC tool. While we are far from a production tool, the results included in this report give us confidence that we are on the right path to constructing an interceding code between the plethora of structured grid homogenized assembly solvers and the supercomputer consuming PROTEUS-SN code. With time, the creation of a suitable preconditioner along with typical optimization should reduce the performance shortcomings that are observable today.

Marin-Lafleche, A.↗

Automatic Differentiation for Adjoint Stencil Loops

Stencil loops are a common motif in computations including convolutional neural networks, structured-mesh solvers for partial differential equations, and image processing. Stencil loops are easy to parallelise, and their fast execution is aided by compilers, libraries, and domain-specific languages. Reverse-mode automatic differentiation, also known as algorithmic differentiation, autodiff, adjoint differentiation, or back-propagation, is sometimes used to obtain gradients of programs that contain stencil loops. Unfortunately, conventional automatic differentiation results in a memory access pattern that is not stencil-like and not easily parallelisable. In this paper we present a novel combination of automatic differentiation and loop transformations that preserves the structure and memory access pattern of stencil loops, while computing fully consistent derivatives. The generated loops can be parallelised and optimised for performance in the same way and using the same tools as the original computation. We have implemented this new technique in the Python tool PerforAD, which we release with this paper along with test cases derived from seismic imaging and computational fluid dynamics applications.

Automatic Differentiation↗

RELAP-7 Level 2 Milestone Report: Demonstration of a Steady State Single Phase PWR Simulation with RELAP-7

The document contains the simulation results of a steady state model PWR problem with the RELAP-7 code. The RELAP-7 code is the next generation nuclear reactor system safety analysis code being developed at Idaho National Laboratory (INL). The code is based on INL's modern scientific software development framework - MOOSE (Multi-Physics Object-Oriented Simulation Environment). This report summarizes the initial results of simulating a model steady-state single phase PWR problem using the current version of the RELAP-7 code. The major purpose of this demonstration simulation is to show that RELAP-7 code can be rapidly developed to simulate single-phase reactor problems. RELAP-7 is a new project started on October 1 st , 2011. It will become the main reactor systems simulation toolkit for RISMC (Risk Informed Safety Margin Characterization) and the next generation tool in the RELAP reactor safety/systems analysis application series (the replacement for RELAP5). The key to the success of RELAP-7 is the simultaneous advancement of physical models, numerical methods, and software design while maintaining a solid user perspective. Physical models include both PDEs (Partial Differential Equations) and ODEs (Ordinary Differential Equations) and experimental based closure models. RELAP-7 will eventually utilize well posed governing equations for multiphase flow, which can be strictly verified. Closure models used in RELAP5 and newly developed models will be reviewed and selected to reflect the progress made during the past three decades. RELAP-7 uses modern numerical methods, which allow implicit time integration, higher order schemes in both time and space, and strongly coupled multi-physics simulations. RELAP-7 is written with object oriented programming language C++. Its development follows modern software design paradigms. The code is easy to read, develop, maintain, and couple with other codes. Most importantly, the modern software design allows the RELAP-7 code to evolve with time. RELAP-7 is a MOOSE-based application. MOOSE (Multiphysics Object-Oriented Simulation Environment) is a framework for solving computational engineering problems in a well-planned, managed, and coordinated way. By leveraging millions of lines of open source software packages, such as PETSC (a nonlinear solver developed at Argonne National Laboratory) and LibMesh (a Finite Element Analysis package developed at University of Texas), MOOSE significantly reduces the expense and time required to develop new applications. Numerical integration methods and mesh management for parallel computation are provided by MOOSE. Therefore RELAP-7 code developers only need to focus on physics and user experiences. By using the MOOSE development environment, RELAP-7 code is developed by following the same modern software design paradigms used for other MOOSE development efforts. There are currently over 20 different MOOSE based applications ranging from 3-D transient neutron transport, detailed 3-D transient fuel performance analysis, to long-term material aging. Multi-physics and multiple dimensional analyses capabilities can be obtained by coupling RELAP-7 and other MOOSE based applications and by leveraging with capabilities developed by other DOE programs. This allows restricting the focus of RELAP-7 to systems analysis-type simulations and gives priority to retain and significantly extend RELAP5's capabilities.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗