Results for “parallel codes” — Page 16

DOE OSTI2022

Geologic Disposal Safety Assessment (GDSA) Biosphere Model Development

The Spent Fuel and Waste Science and Technology Campaign of the U.S. Department of Energy Office of Nuclear Energy, Office of Spent Fuel and Waste Disposition is conducting research and development on geologic disposal of spent nuclear fuel and high-level nuclear waste. This work includes the Geologic Disposal Safety Assessment (GDSA) program which is charged with development of generic deep geologic repository concepts and system performance assessment models. One part of the GDSA framework is the development of a biosphere model capable of assessing doses to potential receptors exposed to radionuclides released from geologic disposal sites. As part of the GDSA framework, a biosphere model compatible with the PFLOTRAN massively parallel subsurface flow and reactive transport code is under development. The PFLOTRAN model provides the radionuclide source term for the biosphere model. The GDSA Biosphere model then assesses the potential movement of radionuclides through the surface biosphere and the subsequent exposure to a human receptor living in the biosphere. The biosphere model includes pathways originating from the groundwater as well as pathways originating from surface water bodies that have a water exchange with a contaminated groundwater body. The pathways for human exposure include consumption of drinking water, irrigated crops, meat animals, aquatic vegetation, and animals, etc.; external exposure from irrigated ground surfaces, surface water bodies, recreational activities, etc.; and inadvertent exposures such as ingestion of contaminated soils or shower water, etc. The GDSA Biosphere model was designed to be flexible and generic in order to accommodate a variety of different sites and climate states. This presentation will present the on the purpose, design, and development progress of the GDSA Biosphere Model.

GDSA, biosphere, repository↗

DOE OSTI2023

Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems

Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.

Schwartz, Peter↗

DOE OSTI2021

The JOREK non-linear extended MHD code and applications to large-scale instabilities and their control in magnetically confined fusion plasmas

JOREK is a massively parallel fully implicit non-linear extended magneto-hydrodynamic (MHD) code for realistic tokamak X-point plasmas. It has become a widely used versatile simulation code for studying large-scale plasma instabilities and their control and is continuously developed in an international community with strong involvements in the European fusion research programme and ITER organization. This article gives a comprehensive overview of the physics models implemented, numerical methods applied for solving the equations and physics studies performed with the code. A dedicated section highlights some of the verification work done for the code. A hierarchy of different physics models is available including a free boundary and resistive wall extension and hybrid kinetic-fluid models. The code allows for flux-surface aligned iso-parametric finite element grids in single and double X-point plasmas which can be extended to the true physical walls and uses a robust fully implicit time stepping. Particular focus is laid on plasma edge and scrape-off layer (SOL) physics as well as disruption related phenomena. Among the key results obtained with JOREK regarding plasma edge and SOL, are deep insights into the dynamics of edge localized modes (ELMs), ELM cycles, and ELM control by resonant magnetic perturbations, pellet injection, as well as by vertical magnetic kicks. Also ELM free regimes, detachment physics, the generation and transport of impurities during an ELM, and electrostatic turbulence in the pedestal region are investigated. Regarding disruptions, the focus is on the dynamics of the thermal quench (TQ) and current quench triggered by massive gas injection and shattered pellet injection, runaway electron (RE) dynamics as well as the RE interaction with MHD modes, and vertical displacement events. Also the seeding and suppression of tearing modes (TMs), the dynamics of naturally occurring TQs triggered by locked modes, and radiative collapses are being studied.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2022

TOUGH3-FLAC3D: a modeling approach for parallel computing of fluid flow and geomechanics

The recent development of the TOUGH3 code allows for a faster and more reliable fluid flow simulator. At the same time, new versions of FLAC3D are released periodically, allowing for new features and faster execution. In this paper, we present the first implementation of the coupling between TOUGH3 and FLAC3Dv6/7, maintaining parallel computing capabilities for the coupled fluid flow and geomechanical codes. We compare the newly developed version with analytical solutions and with the previous approach, and provide some performance analysis on different meshes and varying the number of running processors. Finally, we present two case studies related to fault reactivation during CO 2 sequestration and nuclear waste disposal. The use of parallel computing allows for meshes with a larger number of elements, and hence more detailed understanding of thermo-hydro-mechanical processes occurring at depth.

58 GEOSCIENCES↗

DOE OSTI2024

The inviscid incompressible limit of Kelvin–Helmholtz instability for plasmas

The Kelvin–Helmholtz Instability (KHI) is an interface instability that develops between two fluids or plasmas flowing with a common shear layer. KHI occurs in astrophysical jets, solar atmosphere, solar flows, cometary tails, planetary magnetospheres. Two applications of interest, encompassing both space and fusion applications, drive this study: KHI formation at the outer flanks of the Earth’s magnetosphere and KHI growth from non-uniform laser heating in magnetized direct-drive implosion experiments. Here, we study 2D KHI with or without a magnetic field parallel to the flow. We use both the GAMERA code, which solves the compressible Euler equations, and the STRATOSPEC code, which solves the Navier-Stokes equations under the Boussinesq approximation, coupled with the magnetic field dynamics. GAMERA is a global three-dimensional MHD code with high-order reconstruction in arbitrary nonorthogonal curvilinear coordinates, which is developed for a large range of astrophysical applications. STRATOSPEC is a three-dimensional pseudo-spectral code with an accuracy of infinite order (no numerical diffusion). Magnetized KHI is a canonical case for benchmarking hydrocode simulations with extended MHD options. An objective is to assess whether or not, and under which conditions, the incompressibility hypothesis allows to describe a dynamic compressible system. For comparing both codes, we reach the inviscid incompressible regime, by decreasing the Mach number in GAMERA, and viscosity and diffusion in STRATOSPEC. Here, we specifically investigate both single-mode and multi-mode initial perturbations, either with or without magnetic field parallel to the flow. The method relies on comparisons of the density fields, 1D profiles of physical quantities averaged along the flow direction, and scale-by-scale spectral densities. We also address the triggering, formation and damping of filamentary structures under varying Mach number or Atwood number, with or without a parallel magnetic field. Comparisons show very satisfactory results between the two codes. The vortices dynamics is well reproduced, along with the breaking or damping of small-scale structures. We end with the extraction of growth rates of magnetized KHI from the compressible regime to the incompressible limit in the linear regime assessing the effects of compressibility under increasing magnetic field. The observed differences between the two codes are explained either from diffusion or non-Boussinesq effects.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2020

Development of MGMC: A proxy Multi-Group Monte Carlo Particle Transport Application

This document details the development and current state of MGMC: a proxy Multi-Group Monte Carlo transport application. The goal of this work was to develop a light-weight Monte Carlo transport solver that could be easily reconfigured to test parallelization strategies targeting advanced architectures and heterogeneous computing environments. Additionally, MGMC is able to act as a proxy-app for testing the development of algorithms and the integration of other libraries and into Monte Carlo transport codes. MGMC is currently able to run in parallel using OpenMP for CPU threads or CUDA for NVIDIA GPUs. MGMC was also used to investigate the using SYCL for portable parallelization targeting either CPU threads, vender-agnostic GPUs, and ARM chips, detailed in Section II. In the process of developing MGMC, several other C++ libraries have been created with the intent for re-use in future research projects and potential inclusion in production-level codes, detailed in Section III. The physics capabilities in MGMC are detailed in Section IV, code verification is detailed in Section V, and performance results are detailed in Section VI.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Data-Driven Optimization of the Processing Window for 316H Components Fabricated Using Laser Powder Bed Fusion

The Advanced Materials and Manufacturing Technologies Program is focused on accelerating the development and deployment of advanced materials and components fabricated via additive manufacturing with a specific focus on laser powder bed fusion (LPBF). As an initial case study, the program has selected 316H stainless steel (SS) as an initial material around which to develop a code case development strategy. This strategy involves two parallel approaches: (1) an equivalency approach whereby round-robin testing across multiple collaborating laboratories demonstrates repeatability in processing and direct comparisons with conventional wrought 316H material and (2) a revolutionary approach to code qualification combining in situ data collection and high-fidelity modeling to capture, predict, and bound the performance of LPBF 316HSS components. As part of this campaign, this work package has initiated an extensive process optimization campaign across three laboratories, each printing variations of LPBF 316HSS using three different LPBF units (Concept Laser, EOS, and Renishaw). In FY23, ORNL has focused on unique experimental designs spanning wide ranges in energy inputs and turning knobs such as scan speed, laser power, hatch spacing, layer thickness, spot size, scan rotation, and more. On the Concept Laser M2, 72 different combinations of processing variables were investigated with duplicate samples and different powder compositions. In total, 252 samples were printed with combined in situ sensing data. A parallel design of experiments was conducted on the Renishaw AM400 with an additional 390 printed specimens for analysis. All 642 miniature specimens, each with unique features included in each print to capture geometry-related heterogeneity, were subjected to high-throughput x-ray computed tomography (XCT) analysis to enable the downselection of specific processing parameters of interest. Then, using electrical discharge machining (EDM), miniature tensile specimens were extracted for mechanical testing and microscopy investigations. From the analysis performed in FY23, it was found that powder composition drastically affects the resulting microstructure and mechanical performance of 316SS. Specifically, changing from 316L to 316HSS powder results in a wide range of grain sizes with varying degrees of preferred grain orientation, which increases as a function of energy density. It was also found that due to stored heat in thin fin–type features, large microstructural differences can be seen within one part printed with one set of processing parameters. These variations in microstructure features, including grain size, the nanoscale dislocation structure, and grain texture, will all affect the irradiation performance and high-temperature mechanical performance of LPBF 316HSS parts. Two sets of concept laser processing parameters, spanning both refined and columnar grain structures, were scaled to print larger 316H builds for campaign testing (high-temperature creep and irradiation). In addition, at least two optimized processing parameter sets were identified for the Renishaw AM400 for round-robin testing in FY24 with Argonne National Laboratory. Future work includes printing samples using identical parameters identified by partner institutions, providing material for corrosion and high-temperature mechanical testing, and continuing evaluations of heterogeneity in larger printed parts.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Data-Driven Optimization of the Processing Window for 316H Components Fabricated Using Laser Powder Bed Fusion

The Advanced Materials and Manufacturing Technologies Program is focused on accelerating the development and deployment of advanced materials and components fabricated via additive manufacturing with a specific focus on laser powder bed fusion (LPBF). As an initial case study, the program has selected 316H stainless steel (SS) as an initial material around which to develop a code case development strategy. This strategy involves two parallel approaches: (1) an equivalency approach whereby round-robin testing across multiple collaborating laboratories demonstrates repeatability in processing and direct comparisons with conventional wrought 316H material and (2) a revolutionary approach to code qualification combining in situ data collection and high-fidelity modeling to capture, predict, and bound the performance of LPBF 316HSS components. As part of this campaign, this work package has initiated an extensive process optimization campaign across three laboratories, each printing variations of LPBF 316HSS using three different LPBF units (Concept Laser, EOS, and Renishaw). In FY23, ORNL has focused on unique experimental designs spanning wide ranges in energy inputs and turning knobs such as scan speed, laser power, hatch spacing, layer thickness, spot size, scan rotation, and more. On the Concept Laser M2, 72 different combinations of processing variables were investigated with duplicate samples and different powder compositions. In total, 252 samples were printed with combined in situ sensing data. A parallel design of experiments was conducted on the Renishaw AM400 with an additional 390 printed specimens for analysis. All 642 miniature specimens, each with unique features included in each print to capture geometry-related heterogeneity, were subjected to high-throughput x-ray computed tomography (XCT) analysis to enable the downselection of specific processing parameters of interest. Then, using electrical discharge machining (EDM), miniature tensile specimens were extracted for mechanical testing and microscopy investigations. From the analysis performed in FY23, it was found that powder composition drastically affects the resulting microstructure and mechanical performance of 316SS. Specifically, changing from 316L to 316HSS powder results in a wide range of grain sizes with varying degrees of preferred grain orientation, which increases as a function of energy density. It was also found that due to stored heat in thin fin–type features, large microstructural differences can be seen within one part printed with one set of processing parameters. These variations in microstructure features, including grain size, the nanoscale dislocation structure, and grain texture, will all affect the irradiation performance and high-temperature mechanical performance of LPBF 316HSS parts. Two sets of concept laser processing parameters, spanning both refined and columnar grain structures, were scaled to print larger 316H builds for campaign testing (high-temperature creep and irradiation). In addition, at least two optimized processing parameter sets were identified for the Renishaw AM400 for round-robin testing in FY24 with Argonne National Laboratory. Future work includes printing samples using identical parameters identified by partner institutions, providing material for corrosion and high-temperature mechanical testing, and continuing evaluations of heterogeneity in larger printed parts.

36 MATERIALS SCIENCE↗

DOE OSTI2023

TReactMech v4.217

TReactMech couples geomechanical processes (poroelasticity, failure, and inelastic strain) with multiphase nonisothermal flow (derived from TOUGH2) and reactive geochemical transport. At its core is the reactive-transport code TOUGHREACT v4.13. TReactMech is an efficient hybrid parallel simulator, solving the geomechanics using finite elements and MPI/PetSc, the multiphase flow using integrated finite difference and MPI/PETSc, and the reactive chemistry using OpenMP. The advantages of TReactMech are in its multiphase flow capabilities (e.g., supercritical CO2, supercritical water, air) and parallel geomechanics including full 3-D stress tensor, shear and tensile failure, coupled to porosity and permeability changes. It is backwardly compatible with TOUGH2 and TOUGHREACT v4.13, allowing for easier transitions between the codes. TReactMech can be used to simulate many natural and engineered subsurface systems, including geothermal reservoirs, borehole heat exchangers, geologic carbon sequestration, geologic storage of nuclear waste, groundwater resources, weathering, sediment diagenesis, seafloor hydrothermal circulation, hydrofracturing in unconventional reservoirs, and injection/production-induced surface deformation.

Sonnenthal, Eric↗

DOE OSTI2019

Fit Fly: A Case Study of Interconnect Innovation through Parallel Simulation

To meet the demand for exascale-level performance from high-performance computing (HPC) interconnects, many system architects are turning to simulation results for accurate and reliable predictions of the performance of prospective technologies. Testing full-scale networks with a variety of benchmarking tools, including synthetic workloads and application traces, can give crucial insight into what ideas are most promising without needing to physically construct a test network. While flexible, however, this approach is extremely compute time intensive. We address this time complexity challenge through the use of large-scale, optimistic parallel simulation that ultimately leads to faster HPC network architecture innovations. In this paper we demonstrate this innovation capability through a real-world network design case study. Specifically, we have simulated and compared four extreme-scale interconnects: Dragonfly, Megafly, Slim Fly, and a new dual-rail-dual-plane variation of the Slim Fly network topology. We present this new variant of Slim Fly, dubbed Fit Fly, to show how interconnect innovation and evaluation-beyond what is possible through analytic methods-can be achieved through parallel simulation. We validate and compare the model with various network designs using the CODES interconnect simulation framework. By running large-scale simulations in a parallel environment, we are able to quickly generate reliable performance results that can help network designers break ground on the next generation of high-performance network designs.

High Performance Computing↗

DOE OSTI2024

Verification of electromagnetic simulation capabilities in global gyrokinetic particle-in-cell code GTS

Recently, the numerical scheme presented by Mishchenko et al. enabled explicit gyrokinetic simulations of low-frequency electromagnetic instabilities in tokamaks at experimentally relevant values of plasma β⁠. This scheme resolved the long-standing cancellation problem that previously hindered gyrokinetic particle-in-cell code simulations of magnetohydrodynamic phenomena with inherently small parallel electric fields. Moreover, the scheme did not employ approximations that eliminate critical tearing-type instabilities. Here, we report on the implementation of this numerical scheme in the global gyrokinetic particle-in-cell code GTS. This implementation allows for a more complete and accurate picture of interaction between small scale turbulence and MHD modes in tokamaks. Additionally, we present a comprehensive set of verification simulations of numerous electromagnetic instabilities relevant to present-day tokamaks. These simulations encompass the kinetic ballooning mode, the internal kink mode, the tearing mode, the micro-tearing mode, and the toroidal Alfven eigenmode destabilized by energetic ions, which are all instrumental in understanding tokamak physics. We will also showcase the preliminary nonlinear simulations of kinetic ballooning instabilities and (2,1) island formation due to tearing mode instability. These simulations validate the accuracy of the scheme implementation and pave the way for studying how these instabilities affect plasma confinement and performance.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2021

Devastator Parallel Discrete Event Simulation Runtime (Devastator) v1.0

The Devastator runtime is a modern C++ implementation of optimistic parallel discrete event simulation methods. Devastator allows simulation application code to productively specify their component and event functionality with C++14 constructs. It utilizes GASNet-EX for distributed memory communication and includes parallel performance optimizations such as light-weight thread message queues and asynchronous GVT. Furthermore, it supports efficient event broadcasts and pause-rewind-resume functionality to support periodic load balancing and outer loop optimization algorithms.

Chan, Cy↗

DOE OSTI2025

Thermonuclear Burn in a Multiphysics Code on GPUs

Multiphysics codes links to a library called SINGE for the calculation of thermonuclear (TN) burn rates, but some current multiphysics codes do not attempt to leverage the support for parallel operation that SINGE provides. Our goal is to investigate implementations of the SINGE workflow and analyze how the use of a performance portability layer could reduce run time on CPU archi tectures while also supporting GPU architectures without requiring code modifications. We looked to the Kokkos C++ Performance Portability Ecosystem to implement hardware agnostic parallel patterns.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2018

Parallel k-means Clustering of Geospatial Data Sets Using Manycore CPU Architectures

The increasing availability of high-resolution geospatiotemporal data sets from sources such as observatory networks, remote sensing platforms, and computational Earth system models has opened new possibilities for knowledge discovery and mining of weather, climate, ecological, and other geoscientific data sets fused from disparate sources. Many of the standard tools used on individual workstations are impractical for the analysis and synthesis of data sets of this size; however, new algorithmic approaches that can effectively utilize the complex memory hierarchies and the extremely high levels of parallelism available in state-of-the-art high-performance computing platforms can enable such analysis. Here, we describe pKluster, an open-source tool we have developed for accelerated k-means clustering of geospatial and geospatiotemporal data, and discuss algorithmic modifications and code optimizations we have made to enable it to effectively use parallel machines based on novel CPU architectures—such as the Intel “Knights Landing” Xeon Phi and Skylake Xeon processors—with many cores and hardware threads, and employing significant single instruction, multiple data (SIMD) parallelism. We outline some applications of the code in ecology and climate science contexts and present a detailed discussion of the performance of the code for one such application, LiDAR-derived vertical vegetation structure classification.

Mills, Richard T↗

DOE OSTI2021

SOMAFOAM: An OpenFOAM based solver for continuum simulations of low-temperature plasmas

Here, we report the development of SOMAFOAM, a finite volume framework for performing continuum simulations of low-temperature plasmas. The primary goal of this work is to discuss the features of SOMAFOAM along with representative results provided as examples for a range of operating conditions and geometries. This includes plasma and plasma–dielectric systems operating in direct current, radio frequency, and microwave regimes from pressures as low as 100 mTorr to atmospheric pressure. The code has several useful features including the ability to run massively parallel simulations using arbitrary geometries, structured/unstructured meshes, choice of models such as drift–diffusion/full-momentum at runtime, and species-dependent timesteps to name a few. The verification/validation studies presented include comparison with previously published continuum simulations (low-pressure direct current and radio frequency plasma), with experiments (Gaseous Electronics Conference Reference Cell and microwave microplasma ignited in a split ring resonator), and previously published kinetic simulations (low-pressure radio frequency plasma). Other examples provided include a direct current atmospheric pressure microplasma bounded by dielectric sidewalls and a helium–nitrogen plasma ignited using a needle electrode facing a dielectric. The performance of the code is also discussed with serial and distributed memory parallel runs demonstrated up to 512 cores. The design and implementation of the code in a modular object-oriented framework allows for easy extension and seamless coupling with other codes and can be expected to play an important role in both academia and industry.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2025

HPC for Optimizing Process Parameters to Control Material Evolution in Seamless Induction Hardening of Wind Turbine Main Shaft Bearings

Work proposed in this project focused on understanding the effect of martensitic transformation in the steel on the potential for cracking during seamless induction hardening (SIH) as a function of process conditions to allow the process to optimally scale up. Large-scale, three-dimensional phase-field simulations of martensitic transformation were performed using MEUMAPPS-SS (Microstructure Evolution Using Massively Parallel Phase-field Simulations – Solid State) code developed at Oak Ridge National Laboratory. The simulations were guided by location-specific thermal history generated by experimental measurements of time-temperature history generated at The Timken Company. The simulations were able to capture the morphological evolution of the martensite variants in an Fe-1.0C-1.5Cr steel based on the Nishiyama-Wasserman (NW) orientation relationship. The simulations were also able to quantify the stress-state at the interface between impinging martensite variants. The simulations indicated that the magnitude of the various stress and strain components were dependent on the sizes of the impinging plates with a reduction in these quantities with reduced plate size in agreement with experimental findings. The results obtained from the simulations will be used to guide the optimization of the alloy thermal conditions to eliminate quench cracking during SIH of bearing steels.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2024

Drift kinetic electrostatic simulations of the edge localized mode heat pulse

In the present work, electrostatic drift kinetic simulations of parallel plasma transport within the tokamak scrape-off layer (SOL) are conducted using the COGENT code. The SOL configuration is represented in one-dimensional slab geometry, incorporating a heat source localized in the midplane. The heat source parameters correspond to those characterizing edge-localized modes observed in the Joint European Torus (JET) tokamak. The numerical model includes kinetic treatment of both ions and electrons, a simplified model for the gyrokinetic Poisson equation that allows one to step over short time scales associated with fast electrostatic shear Alfvèn waves, and the logical sheath boundary condition (LSBC) that enforces global system quasineutrality. A third-order accurate LSBC is derived to be consistent with the third-order accurate upwind advection scheme utilized in the code, and it was shown to noticeably impact the simulation results, especially parallel heat flux at the target plate. The findings of this study are in agreement with results from preceding fluid and kinetic simulations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2025

Performance Evaluation of Different Parallel Programming Models in SCALE-Shift Sequences for Criticality and Shielding Applications [Abstract]

The SCALE code system has been widely used for nuclear criticality safety, reactor physics, radiation shielding, source term generation, and inventory analyses by researchers, industry, and regulatory bodies. Although limited support for shared- and distributed-memory parallel processing was introduced via C++ threading, OpenMP, and MPI, a hybrid parallel programming model with both distributed- and shared-memory parallelism has not been fully supported in the SCALE code system.

Nuclear Criticality Safety Program (NCSP)↗