Investigation of Doppler Broadening Methods Within the Shift Monte Carlo Radiation Transport Code
Explore the source record for details and available documents.
Evans, Thomas↗
Engineering topics
Evans, Thomas
Publications and source records attributed to Evans, Thomas.
Celeritas: Accelerating Geant4 with GPUs
Celeritas [1] is a new Monte Carlo (MC) detector simulation code designed for computationally intensive applications (specifically, High Lumi- nosity Large Hadron Collider (HL-LHC) simulation) on high-performance heterogeneous architectures. In the past two years Celeritas has advanced from prototyping a GPU-based single physics model in infinite medium to implementing a full set of electromagnetic (EM) physics processes in complex geometries. The current release of Celeritas, version 0.3, has incorporated full device-based navigation, an event loop in the presence of magnetic fields, and detector hit scoring. New functionality incorporates a scheduler to offload electromagnetic physics to the GPU within a Geant4-driven simulation, enabling integration of Celeritas into high energy physics (HEP) experimental frameworks such as CMSSW. On the Summit supercomputer, Celeritas performs EM physics between 6 and 32 faster using the machine’s Nvidia GPUs compared to using only CPUs. When running a multithreaded Geant4 ATLAS test beam application with full hadronic physics, using Celeritas to accelerate the EM physics results in an overall simulation speedup of 1.8–2.3× on GPU and 1.2× on CPU.
Johnson, Seth R.↗
A Cast of Thousands: How the IDEAS Productivity Project Has Advanced Software Productivity and Sustainability
Computational and data-enabled science and engineering are revolutionizing advances throughout science and society, at all scales of computing. For example, teams in the U.S. Department of Energy’s Exascale Computing Project have been tackling new frontiers in modeling, simulation, and analysis by exploiting unprecedented exascale computing capabilities—building an advanced software ecosystem that supports next-generation applications and addresses disruptive changes in computer architectures. However, concerns are growing about the productivity of the developers of scientific software. Members of the Interoperable Design of Extreme-scale Application Software project serve as catalysts to address these challenges through fostering software communities, incubating and curating methodologies and resources, and disseminating knowledge to advance developer productivity and software sustainability. This article discusses how these synergistic activities are advancing scientific discovery—mitigating technical risks by building a firmer foundation for reproducible, sustainable science at all scales of computing, from laptops to clusters to exascale and beyond.
97 MATHEMATICS AND COMPUTING↗
Photonuclear Physics in SCALE
Radiation shielding modeling involves a wide range of applications, such as power reactors, critical experiments, portable sources, linear accelerators, and fusion systems. Particle energy ranges vary from thermal energies to fast energy spectra. MAVRIC, as the radiation shielding sequence of the SCALE code system [1] developed and maintained by Oak Ridge National Laboratory, is capable of modeling both neutron and photon interactions in matter and is widely used by the US Nuclear Regulatory Commission, Department of Energy, the National Nuclear Security Administration, and radiation shielding practitioners all around the world. Although most applications don’t involve highly energetic photons that can cause photonuclear reactions and create coupled chains of high energy, ionized electrons and bremsstrahlung radiation within a material medium, when present, these reactions can have a significant impact upon the downstream particle fluxes from incident photons above 20 MeV. For example, linear accelerators used for medical isotope production are examples of applications in which high-energetic photons are generated and used for driving or controlling other components of a complex facility. Radiation shielding analyses of such a facility would require accounting for any and all significant sources of radiation. Unfortunately, MAVRIC included no data and methods to model either photonuclear reactions or the bremsstrahlung photons until this work, which enabled accounting for both. Photonuclear reaction cross sections were imported from the Evaluated Nuclear Data File libraries, and corresponding methods were implemented in the SCALE code system; see Section 2 for how the nuclear data are imported and used. Bremsstrahlung photons were modeled by assuming thick-target approximation whereby all generated electrons and positrons due to photon interactions are assumed to be absorbed within the same material in which they were born, yielding a quick particle transport simulation method by using fast lookup tables for generating subsequent gammas; see Section 4 for details. A suite of verification models was used to test implemented methods and associated data. Tests yielded satisfactory results for the new features. Enhancements will be released as part of the next major SCALE release (v7.0), which is anticipated to be deployed in 2025, with beta releases available in the interim.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Celeritas R&D Report: Accelerating Geant4
Celeritas is a new Monte Carlo (MC) detector simulation code designed for computationally intensive applications on high-performance heterogeneous architectures. In the past two years Celeritas has advanced from prototyping a Graphics Processing Unit (GPU)-based single physics model in infinite medium to implementing a full set of electromagnetic (EM) physics processes in complex geometries. The current release of Celeritas, version 0.4, has incorporated full device-based navigation, an event loop in the presence of magnetic fields, and detector hit scoring. New functionality incorporates a scheduler to offload electromagnetic physics to the GPU within a Geant4-driven simulation, enabling straightforward integration of Celeritas into the high energy physics (HEP) experimental frameworks CMSSW and ATLAS FullSimLight. On the Perlmutter supercomputer, Celeritas performs EM physics between 3× and 18× faster using the machine’s Nvidia GPUs compared to using only CPUs, corresponding to an electrical power efficiency up to a factor of 5. When running a multithreaded Geant4 ATLAS test beam application with full hadronic physics, using Celeritas to accelerate the EM physics results in an overall simulation speedup of 1.7–2.2× on GPU and 1.2× on CPU. In a CMS test application using tt¯ events and the prototype Run 4 configuration, compared to Geant4 CPU, Celeritas with a Nvidia A100 improves overall throughput up to a factor of 2.7× but cannot be efficiently shared with more than 8 cores.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Frontier: Exploring Exascale
As the US Department of Energy (DOE) computing facilities began deploying petascale systems in 2008, DOE was already setting its sights on exascale. In that year, DARPA published a report on the feasibility of reaching exascale. The report authors identified several key challenges in the pursuit of exascale including power, memory, concurrency, and resiliency. That report informed the DOE's computing strategy for reaching exascale. With the deployment of Oak Ridge National Laboratory's Frontier supercomputer, we have officially entered the exascale era. In this paper, we discuss Frontier's architecture, how it addresses those challenges, and describe some early application results from Oak Ridge Leadership Computing Facility's Center of Excellence and the Exascale Computing Project.
Atchley, Scott {Leadership Computing}↗
Exascale Multiphysics Nuclear Reactor Simulations for Advanced Designs
ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.
Hamilton, Steven P.↗
Weight window variance reduction on GPUs in the Shift Monte Carlo Code
Explore the source record for details and available documents.
Royston, Katherine↗
Detector Simulation Challenges for Future Accelerator Experiments
Detector simulation is a key component for studies on prospective future high-energy colliders, the design, optimization, testing and operation of particle physics experiments, and the analysis of the data collected to perform physics measurements. This review starts from the current state of the art technology applied to detector simulation in high-energy physics and elaborates on the evolution of software tools developed to address the challenges posed by future accelerator programs beyond the HL-LHC era, into the 2030–2050 period. New accelerator, detector, and computing technologies set the stage for an exercise in how detector simulation will serve the needs of the high-energy physics programs of the mid 21st century, and its potential impact on other research domains.
43 PARTICLE ACCELERATORS↗
Application Results on Early Exascale Hardware
This Exascale Computing Project (ECP) milestone report summarizes the status of 27 of the 31 ECP Applications Development (AD) subprojects at the end of FY21. In November and December of 2021, a comprehensive assessment of AD projects was conducted by the ECP leadership along with external subject matter experts (SMEs). (NNSA application projects are reviewed separately using the ASC milestone process.) The AD review committee—consisting of the AD lead, AD deputy, Level 3 (L3), and at least one external project SME—was tasked with evaluating each project’s progress relative to ECP project goals specified in the FY21 timeline. Key areas of focus were code maturity and performance on pre-exascale systems, an in-depth analysis of final key performance parameter (KPP) verification contracts, and future R&D priorities in the final year of ECP and beyond. As such, this report contains not only an accurate snapshot of each subproject’s current status but also represents a broad account of successes and challenges in porting large scientific applications to DOE’s next-generation high-performance computing architectures – the Frontier and Aurora systems.
97 MATHEMATICS AND COMPUTING↗
Validation of Light Water Reactor Ex-Core Calculations with VERA
The Consortium for Advanced Simulation of Light Water Reactors (CASL) Virtual Environment for Reactor Applications (VERA) is a reactor simulation software. It offers unique capabilities by combining high-fidelity in-core radiation transport with temperature feedback by using MPACT (a deterministic neutron transport code) and COBRA-TF (a thermal-hydraulic code) with follow-on, fixed-source transport calculations using the Shift Monte Carlo code to calculate ex-core quantities of interest. In these coupled calculations, MPACT provides Shift with the fission source for follow-on ex-core calculations. These ex-core simulations can be set up to calculate detector responses, as well as the flux and fluence in ex-core regions of interest, such as the reactor pressure vessel, nozzle, and irradiated capsules. A Watts Bar Nuclear Plant Unit 1 (WBN1) ex-core model was developed, as described in this paper, and this model was used to perform coupon calculations. The results for the coupon flux calculations show close agreement with the reference values for cycle 1 produced by the two-dimensional Discrete Ordinates Transport (DORT) code and presented in a BWXT Services Inc. report. However, differences in the results (10%) seen in cycles 2 and 3 and the reasons for these differences are discussed in this paper. The VERA WBN1 model was also used to perform a vessel fluence calculation for cycle 1. Additionally, a collaboration between CASL and Duke Energy led to the first code-to-code validation of VERA for reactor ex-core applications that used a model for the Shearon Harris reactor. Results from this collaboration show excellent agreement between VERA and the Monte Carlo N-Particle Transport Code for the detector response calculations. The work performed under this collaboration is also detailed in this paper.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Algorithm for Free Gas Elastic Scattering without Rejection Sampling
Explore the source record for details and available documents.
Biondo, Elliott↗
Free Gas Elastic Scattering Algorithm for GPU-Based Particle Tracking
Explore the source record for details and available documents.
Biondo, Elliott↗
Novel features and GPU performance analysis for EM particle transport in the Celeritas code
Celeritas is a new computational transport code designed for high-performance simulation of high-energy physics detectors. This work describes some of its current capabilities and the design choices that enable the rapid development of efficient on-device physics. The abstractions that underpin the code design facilitate low-level performance tweaks that require no changes to the higher-level physics code. We evaluate a set of independent changes that together yield an almost 40% speedup over the original GPU code for a net performance increase of 220× for a single GPU over a single CPU running 8.4M tracks on a small demonstration physics app.
Johnson, Seth R.↗
Performance Portability in the Exascale Computing Project: Exploration Through a Panel Series
Performance portability is a critical issue for the Exascale Computing Project (ECP) because of nontrivial architectural differences between machines available today and those expected at exascale. Many ECP project teams are working toward performance portability, and would expect to benefit from sharing lessons learned, identifying gaps, and discovering opportunities for partnerships. To facilitate this communication, the IDEAS-ECP project partnered with the three focus areas of ECP (application development, software technology, and hardware and integration), and Department of Energy computing facilities, to lead a series of panel discussions on performance portability. The panels were organized around broadly common themes of algorithmic and data locality challenges. In this article, we describe the panel series, its objectives, and perspectives from the various areas of the project. Finally, we also discuss use cases that are distinctive, as well as conclusions drawn from the collective experience of the participants.
97 MATHEMATICS AND COMPUTING↗
Map Applications to Target Exascale Architecture with Machine-Specific Performance Analysis, Including Challenges and Projections
This Exascale Computing Project (ECP) milestone report summarizes the status of all 30 ECP Applications Development (AD) subprojects at the end of FY20. In October and November of 2020, a comprehensive assessment of AD projects was conducted by the ECP leadership. Reviews occurred virtually between October 27, 2020 and November 12, 2020. The review committee—consisting of the AD lead, deputy, and L3—was tasked with evaluating each subproject’s progress in porting their codes to early exascale architectures considered precursors to the planned exascale machines. This includes characterizing which modules have been ported to multi-accelerator nodes, initial performance analyses, the status of software integration, and a current vision of successes, obstacles, and next steps. As such, this report contains not only an accurate snapshot of each subproject’s current status but also represents an unprecedentedly broad account of experiences in porting large scientific applications to next-generation high-performance computing architectures.
97 MATHEMATICS AND COMPUTING↗
As-Built Simulation of the High Flux Isotope Reactor
The Oak Ridge National Laboratory High Flux Isotope Reactor (HFIR) is an 85 MWt flux trap-type research reactor that supports key research missions, including isotope production, materials irradiation, and neutron scattering. The core consists of an inner and an outer fuel element containing 171 and 369 involute-shaped plates, respectively. The thin fuel plates consist of a U 3 O 8 -Al dispersion fuel (highly enriched), an aluminum-based filler, and aluminum cladding. The fuel meat thickness is varied across the width of the involute plate to reduce thermal flux peaks at the radial edges of the fuel elements. Some deviation from the designed fuel meat shaping is allowed during manufacturing. A homogeneity scan of each fuel plate checks for potential anomalies in the fuel distribution by scanning the surface of the plate and comparing the attenuation of the beam to calibration standards. While typical HFIR simulations use homogenized fuel regions, explicit models of the plates were developed under the Low-Enriched Uranium Conversion Program. These explicit models typically include one inner and one outer fuel plate with nominal fuel distributions, and then the plates are duplicated to fill the space of the corresponding fuel element. Therefore, data extracted from these simulations are limited to azimuthally averaged quantities. To determine the reactivity and physics impacts of an as-built outer fuel element and generate azimuthally dependent data in the element, 369 unique fuel plate models were generated and positioned. This model generates the three-dimensional (i.e., radial–axial–azimuthal) plate power profile, where the azimuthal profile is impacted by features within the adjacent control element region and beryllium reflector. For an as-built model of the outer fuel element, plate-specific homogeneity data, 235 U loading, enrichment, and channel thickness measurements were translated into the model, yielding a much more varied azimuthal power profile encompassed by uncertainty factors in analyses. These models were run with the ORNL-TN and Shift Monte Carlo tools, and they contained upwards of 500,000 cells and 100,000 unique tallies.
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
Secondary-Source Core Reload Modeling with VERA
The CASL reactor simulation package VERA has been adapted to provide high-fidelity simulation capabilities for modeling source range detector response during subcritical reactor configurations. New features include the activation and shuffling of secondary-source assemblies, use of burned fuel neutron emission data from the ORIGEN depletion solver to the MPACT deterministic neutron transport solver, allowance of user-defined sources in MPACT based on material composition, ability to solve the subcritical source-driven system with neutron multiplication using the MPACT diffusion solver, and transfer of the calculated fission source from MPACT to the continuous-energy Monte Carlo solver Shift for final detector response evaluation using the CADIS methodology for variance reduction. These new capabilities were benchmarked against Watts Bar Unit 1 plant operating data for the first few fuel loading steps and were found to demonstrate excellent agreement with the measured data.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗