SEARCH · Search NASA

Results for “parallel codes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

DOE OSTI2021

All-Atom Simulation of 3D Hot Spot Formation in Shocked TATB Explosive

TATB is an insensitive high explosive (IHE) critical to the stockpile that is challenging to model at the continuum scale. Advanced detonation models in the Cheetah high explosive chemistry code require validation though subscale simulations. High explosive initiation is determined by micron-scale physics of hot spots formed a shock-collapsed pores. Pore sizes between 100 nm and 1 μm are believed to be the most important for determining the shock sensitivity of TATB. This range of pore sizes is difficult to access at the atomic scale through allatom molecular dynamics (MD) simulations, even with Sierra-class computers. Quasi-2D simulations are widely used and allow much larger pore sizes (up to 400 nm) to be studied, but the applicability of 2D simulations to the actual 3D pore response is not understood. Resolving these uncertainties through “full physics” MD modeling is key for generalizing, parameterizing, and validating the kinds of continuum models used to inform design, safety, and performance. This work was a continuation of FY20 efforts pushing simulations to full 3D with the largest-ever all-atom simulations of an explosive. These were the first all-atom full-3D simulations of large hot spots thought to govern explosive detonation and required over a billion atoms. Simulations were performed using LAMMPS, an open SNL science code. MD explosive models present unique challenges, even for established codes such as LAMMPS. Their model forms are more complex than typical models for metals, while simulating high temperature-pressure conditions is demanding and increases computational cost. Scaling problems in GPU-enabled MD algorithms initially limited simulations to <100 million atoms but were resolved through collaboration with SNL. An overall 24x speedup was obtained relative to CPU machines. Specialized analysis of these simulations required a bottom-up refactoring and algorithm parallelization of in-house codes and application of computer vision algorithms to extract meaningful information.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Massively parallel modeling and inversion of electrical resistivity tomography data using PFLOTRAN

Abstract. Electrical resistivity tomography (ERT) is a broadly accepted geophysical method for subsurface investigations. Interpretation of field ERT data usually requires the application of computationally intensive forward modeling and inversion algorithms. For large-scale ERT data, the efficiency of these algorithms depends on the robustness, accuracy, and scalability on high-performance computing resources. In this regard, we present a robust and highly scalable implementation of forward modeling and inversion algorithms for ERT data. The implementation is publicly available and developed within the framework of PFLOTRAN, an open-source, state-of-the-art massively parallel subsurface flow and transport simulation code. The forward modeling is based on a finite-volume discretization of the governing differential equations, and the inversion uses a Gauss–Newton optimization scheme. To evaluate the accuracy of the forward modeling, two examples are first presented by considering layered (1D) and 3D earth conductivity models. The computed numerical results show good agreement with the analytical solutions for the layered earth model and results from a well-established code for the 3D model. Inversion of ERT data, simulated for a 3D model, is then performed to demonstrate the inversion capability by recovering the conductivity of the model. To demonstrate the parallel performance of PFLOTRAN's ERT process model and inversion capabilities, large-scale scalability tests are performed by using up to 131 072 processes on a leadership class supercomputer. These tests are performed for the two most computationally intensive steps of the ERT inversion: forward modeling and Jacobian computation. For the forward modeling, we consider models with up to 122 ×106 degrees of freedom (DOFs) in the resulting system of linear equations and demonstrate that the code exhibits almost linear scalability on up to 10 000 DOFs per process. On the other hand, the code shows superlinear scalability for the Jacobian computation, mainly because all computations are fairly evenly distributed over each process with no parallel communication.

58 GEOSCIENCES↗

DOE OSTI2019

Extracting SIMD Parallelism from Recursive Task-Parallel Programs

The pursuit of computational efficiency has led to the proliferation of throughput-oriented hardware, from GPUs to increasingly wide vector units on commodity processors and accelerators. This hardware is designed to efficiently execute data-parallel computations in a vectorized manner. However, many algorithms are more naturally expressed as divide-and-conquer, recursive, task-parallel computations. In the absence of data parallelism, it seems that such algorithms are not well suited to throughput-oriented architectures. This paper presents a set of novel code transformations that expose the data parallelism latent in recursive, task- parallel programs. These transformations facilitate straightforward vectorization of task-parallel programs on commodity hardware. We also present scheduling policies that maintain high utilization of vector resources while limiting space usage. Across several task-parallel benchmarks, we demonstrate both efficient vector resource utilization and substantial speedup on chips using Intel’s SSE4.2 vector units, as well as accelerators using Intel’s AVX512 units. We then show through rigorous sampling that, in practice, our vectorization techniques are effective for a much larger class of programs.

Recursive Programs, task parallelism, vectorizatio↗

DOE OSTI2020

NWChem: Past, present, and future

Specialized computational chemistry packages have permanently reshaped the landscape of chemical sciences by providing tools to support and guide the experimental effort and for prediction of chemical and materials properties. In this regard, a special role has been played by electronic structure packages where complex chemical and materials processes can be modeled using first-principle-driven methodologies. Over the last few decades, the rapid development of computing technologies and a tremendous increase in computational power has offered a unique chance to study complex chemical transformations using sophisticated and predictive many-body techniques to describe correlated behavior of electrons in molecular and condensed phase systems at different levels of theory. In enabling these simulations, a critical role has been played by novel parallel algorithms capable of taking advantage of computational resources to address polynomial scaling of electronic structure methods. NWChem was among the first electronic structure codes that focused on delivering scalable performance for electronic structure simulations. Herein, we briefly review the NWChem suite of computational codes including its history, design principles, parallel tools, current capabilities, outreach and outlook.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2024

Initial Development of Fusion Magnet Simulation Capabilities for Performance and Safety Evaluation Using the MOOSE Framework

Fusion energy holds the promise of being a transformative technology as a carbon-neutral, sustainable source of energy. Whole device modeling and the development of fusion digital twins will be increasingly important for emerging fusion device concepts at both national laboratories and within the commercial fusion industry. However, meeting the challenge of whole device modeling of fusion energy devices requires robust, multiphysics, multiscale modeling and simulation technologies capable of running on large-scale supercomputers. Detailed analysis of individual systems at-scale is also required to ensure safe and efficient operation as well as provide the safety basis for future device designs and licensing activities. In a tokamak, toroidal and poloidal magnets confine and shape the fusion plasma to promote the fusion reaction. High plasma temperatures and high magnetic field requirements in modern design concepts (leading to high amounts of energy stored within each magnet) impose electrical, thermal, and mechanical loads on the magnet components, which in turn impacts the safety considerations of the magnet and their supporting systems. Idaho National Laboratory (INL) has a history of working in this space, including development and benchmarking of the Magnetic System Circuitry Analysis Program (MSCAP) and Magnet Arcing (MAGARC) codes to study magnet quench events; notably, MAGARC was used to study quenching during the ITER Engineering Design Activity. However, these legacy codes and capabilities are not parallel and scalable, and new tools are required for future advances in this area, which leads to the INL-developed Multiphysics Object-Oriented Simulation Environment (MOOSE) framework. Developed originally for fission reactor systems under United States Department of Energy, Office of Nuclear Energy modeling and simulation programs, the MOOSE framework has been well-suited to multiscale, multiphysics modeling and simulation needs for nuclear systems. The framework is open-source, well-tested, under continuous development and deployment, and developed to a Nuclear Quality Assurance, Level 1 software quality standard. MOOSE has also been used in the fusion space previously in several projects: INL’s Tritium Migration Analysis Program, Version 8 (TMAP8) for tritium migration, UK Atomic Energy Authority’s A Unified Resource for OpenMC (fusion) Reactor Applications (AURORA) code for fusion thermo-mechanical and neutronics analysis, and Argonne National Laboratory’s Cardinal for high-fidelity computational fluid dynamics and neutronics. However, to model superconducting magnets, several MOOSE enhancements are required: additions to the current MOOSE electromagnetic capabilities, new material libraries for superconductors of interest (such as YBCO), as well as fusion-specific models for thermo-mechanics. This talk will discuss initial development activities to build these capabilities in MOOSE, focusing on initial validation and benchmarking activities. Proposed coupling workflows and future work to support the simulation of fusion magnets and magnet structural assemblies for performance and safety evaluation in MOOSE will also be discussed.

70 - PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2022

Spinor $GW$ Bethe-Salpeter calculations in BerkeleyGW: Implementation, symmetries, benchmarking, and performance

Computing the GW quasiparticle band structure and Bethe-Salpeter equation (BSE) absorption spectra for materials with spin-orbit coupling have commonly been done by treating GW corrections and spin-orbit coupling (SOC) as separate perturbations to density-functional theory. However, accurate treatment of materials with strong spin-orbit coupling (such as many topological materials of recent interest, and thermoelectrics) often requires a nonperturbative approach using spinor wave functions in the Kohn-Sham equation and GW/BSE. Such calculations have only recently become available, in particular for the BSE. Here, we have implemented this approach in the plane-wave pseudopotential GW/BSE code BerkeleyGW, which is highly parallelized and widely used in the electronic-structure community. We present reference results for quasiparticle band structures and optical absorption spectra of solids with different strengths of spin-orbit coupling, including Si, Ge, GaAs, GaSb, CdSe, Au, and Bi 2 Se 3 . The calculated quasiparticle band gaps of these systems are found to agree with experiment to within a few tens of meV. SOC splittings are found to be generally in better agreement with experiment, including quasiparticle corrections to band energies. The absorption spectrum of GaAs is not significantly impacted by the inclusion of spin-orbit coupling due to its relatively small value (0.2 eV) in the Λ direction, while the absorption spectrum of GaSb calculated with the spinor GW/BSE captures the large spin-orbit splitting of peaks in the spectrum. For the prototypical topological insulator Bi 2 Se 3 , we find a drastic change in the low-energy band structure compared to that of DFT, with the spinorial treatment of the GW approximation correctly capturing the parabolic nature of the valence and conduction bands after including off-diagonal self-energy matrix elements. We present the detailed methodology, approach to spatial symmetries for spinors, comparison against other codes, and performance compared to spinless GW/BSE calculations and perturbative approaches to SOC. This work aims to spur further development of spinor GW/BSE methodology in excited-state research software and enables a more accurate and detailed exploration of electronic and optical properties of materials containing elements with large atomic numbers.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2020

PyCDFT: A Python package for constrained density functional theory

In this paper, we present PyCDFT, a Python package to compute diabatic states using constrained density functional theory (CDFT). PyCDFT provides an object-oriented, customizable implementation of CDFT, and allows for both single-point self-consistent-field calculations and geometry optimizations. PyCDFT is designed to interface with existing density functional theory (DFT) codes to perform CDFT calculations where constraint potentials are added to the Kohn–Sham Hamiltonian. Here, we demonstrate the use of PyCDFT by performing calculations with a massively parallel first-principles molecular dynamics code, Qbox, and we benchmark its accuracy by computing the electronic coupling between diabatic states for a set of organic molecules. We show that PyCDFT yields results in agreement with existing implementations and is a robust and flexible package for performing CDFT calculations. The program is available at https://dx.doi.org/10.5281/zenodo.3821097.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2021

Multigrid reduction in time with Richardson extrapolation

The advent of exascale computing will leave many users with access to more computational resources than they can simultaneously use, e.g., billion-way parallelism. In particular, this is true for time-dependent simulations that limit parallelism to the spatial domain. One method to add parallelism in time to existing simulation codes and thus take advantage of ever larger compute resources is Multigrid Reduction in Time (MGRIT). The goal is to achieve a smaller time-to-solution through parallelism in time. In this paper, MGRIT is enhanced with Richardson extrapolation in a cost-efficient way to produce a parallel-in-time method with improved accuracy. Overall, this leads to a large improvement in the accuracy per computational cost of MGRIT.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Charon User Manual (V.2.1) (Rev.01)

This manual gives usage information for the Charon semiconductor device simulator. Charon was developed to meet the modeling needs of Sandia National Laboratories and to improve on the capabilities of the commercial TCAD simulators; in particular, the additional capabilities are running very large simulations on parallel computers and modeling displacement damage and other radiation effects in significant detail. The parallel capabilities are based around the MPI interface which allows the code to be ported to a large number of parallel systems, including linux clusters and proprietary "big iron" systems found at the national laboratories and in large industrial settings.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2022

Charon User Manual: v. 2.2 (revision1)

This manual gives usage information for the Charon semiconductor device simulator. Charon was developed to meet the modeling needs of Sandia National Laboratories and to improve on the capabilities of the commercial TCAD simulators; in particular, the additional capabilities are running very large simulations on parallel computers and modeling displacement damage and other radiation effects in significant detail. The parallel capabilities are based around the MPI interface which allows the code to be ported to a large number of parallel systems, including linux clusters and proprietary “big iron” systems found at the national laboratories and in large industrial settings.

42 ENGINEERING↗

DOE OSTI2019

MPI + OpenACC: Accelerating radiation transport mini-application, minisweep, on heterogeneous systems

According to the article, architectures are rapidly evolving, and exascale machines are expected to offer billion-way concurrency. We need to rethink algorithms, languages and programming models among other components in order to migrate large scale applications and explore parallelism on these machines. Although directive-based programming models allow programmers to worry less about programming and more about science, expressing complex parallel patterns in these models can be a daunting task especially when the goal is to match the performance that the hardware platforms can offer. One such pattern is wavefront. This paper extensively studies a wavefront-based miniapplication for Denovo, a production code for nuclear reactor modeling. We parallelize the Koch–Baker–Alcouffe (KBA) parallel-wavefront sweep algorithm in the main kernel of Minisweep (the miniapplication) using CUDA 9.0, OpenMP 4.0 (SIMD) and OpenACC 2.6. Our OpenACC implementation running on NVIDIA’s next-generation Volta GPU boasts an 85.06x speedup over serial code, which is larger than CUDA’s 83.72x speedup over the same serial implementation. We also explore the scalability of our solution using MPI to decompose our simulation domain, allowing us to run on many nodes and accelerators present in state-of-the-art HPC systems. Our parallelization effort across platforms also motivated us to define an abstract parallelism model that is architecture independent, with a goal of creating software abstractions that can be used by applications employing the wavefront sweep motif.

Computer Science↗

DOE OSTI2021

Toward exascale whole-device modeling of fusion devices: Porting the GENE gyrokinetic microturbulence code to GPU

GENE solves the five-dimensional gyrokinetic equations to simulate the development and evolution of plasma microturbulence in magnetic fusion devices. The plasma model used is close to first principles and computationally very expensive to solve in the relevant physical regimes. In order to use the emerging computational capabilities to gain new physics insights, several new numerical and computational developments are required. Here, we focus on the fact that it is crucial to efficiently utilize GPUs (graphics processing units) that provide the vast majority of the computational power on such systems. In this paper, we describe the various porting approaches considered and given the constraints of the GENE code and its development model, justify the decisions made, and describe the path taken in porting GENE to GPUs. We introduce a novel library called gtensor that was developed along the way to support the process. Performance results are presented for the ported code, which in a single node of the Summit supercomputer achieves a speed-up of almost 15× compared to running on central processing unit (CPU) only. Typical GPU kernels are memory-bound, achieving about 90% of peak. Our analysis shows that there is still room for improvement if we can refactor/fuse kernels to achieve higher arithmetic intensity. We also performed a weak parallel scalability study, which shows that the code runs well on a massively parallel system, but communication costs start becoming a significant bottleneck.

Germaschewski, K. (ORCID:0000000284956354)↗

DOE OSTI2018

Annual Reports in Computational Chemistry, Chapter 3: GronOR Nonorthogonal Configuration Interaction Calculations at Exascale

This chapter describes the implementation and benchmarking of a new software code for nonorthogonal configuration interaction calculations. GronOR is named after the University of Groningen and Oak Ridge National Laboratory, the two institutions that are collaborating on the development of this code for the largest massively parallel supercomputers for open science. The basic implementation approach is described and preliminary benchmarking results are given as obtained from early access to Summit, the next Oak Ridge Leadership Computing Facility's supercomputer with 200 PFlop.

Chemistry↗

DOE OSTI2022

Three-dimensional Skyrme Hartree-Fock-Bogoliubov solver in coordinate-space representation

The coordinate-space representation of the Hartree-Fock-Bogoliubov theory is the method of choice to study weakly bound nuclei whose properties are affected by the quasiparticle continuum space. To describe such systems, we developed a three-dimensional Skyrme-Hartree-Fock-Bogoliubov solver HFBFFT based on the existing, highly optimized and parallelized Skyrme-Hartree-Fock code Sky3D. The code does not impose any self-consistent spatial symmetries such as mirror inversions or parity. The underlying equations are solved in HFBFFT directly in the canonical basis using the fast Fourier transform. To remedy the problems with pairing collapse, we implemented the soft energy cutoff and pairing annealing. The convergence of HFB solutions was improved by a sub-iteration method. The Hermiticity violation of differential operators brought by Fourier-transform-based differentiation has also been solved. Furthermore, the accuracy and performance of HFBFFT were tested by benchmarking it against other HFB codes, both spherical and deformed, for a set of nuclei, both well-bound and weakly-bound.

3D coordinate-space representation↗

DOE OSTI2021

DART-PFLOTRAN: An ensemble-based data assimilation system for estimating subsurface flow and transport model parameters

Ensemble-based Data Assimilation (EDA), based on the Monte Carlo approach, has been effectively applied to estimate model parameters through inverse modeling in subsurface flow and transport problems. However, implementation of EDA approach involves a complicated workflow that include setting up and executing ensemble forward model simulations, processing observations and model simulation results for parameter updates, and repeat for sequential or iterative EDA. To facilitate the management of such workflow and lower the barriers for adopting EDA-based parameter estimation in subsurface science, we develop a generic software frame-work linking the Data Assimilation Research Testbed (DART) with a massively parallel subsurface FLOw and TRANsport code PFLOTRAN. The new DART-PFLOTRAN leverages both the core data assimilation engines in DART and the computational power afforded by PFLOTRAN. In addition to the standard smoother and filtering options, DART-PFLOTRAN enables an iterative EDA workflow based on the Ensemble Smoother for Multiple Data Assimilation method (ES-MDA) to improve estimation accuracy for nonlinear forward problems. Here, we verify the implementation of ES-MDA in DART-PFLOTRAN using two synthetic cases designed to estimate static permeability and dynamic exchange fluxes across the riverbed, respectively, from continuous temperature measurements made across a depth profile. One-dimensional hydro-thermal simulations are performed in both cases to relate temperature responses with the parameters of interest. In the case of estimating dynamic parameters, we demonstrate the flexibility of DART-PFLOTRAN in automating sequential ES-MDA workflow, which will significantly reduce the time researchers spend on managing complex workflows in similar applications. Both studies yield accurate estimations of the parameters compared to their synthetic truth, while ES-MDA leads to more accurate estimation when a high level of nonlinearity exist between observed responses and unknown parameters. With a code base in Python and Fortran, DART-PFLOTRAN paves the way for applications in large-scale subsurface inverse modeling by automating the complex workflow of sequential ES-MDA that can be executed on various computing platforms.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

High-fidelity wind farm simulation methodology with experimental validation

The complexity and associated uncertainties involved with atmospheric-turbine-wake interactions produce challenges for accurate wind farm predictions of generator power and other important quantities of interest (QoIs), even with state-of-the-art high-fidelity atmospheric and turbine models. A comprehensive computational study was undertaken with consideration of simulation methodology, parameter selection, and mesh refinement on atmospheric, turbine, and wake QoIs to identify capability gaps in the validation process. For neutral atmospheric boundary layer conditions, the massively parallel large eddy simulation (LES) code Nalu-Wind was used to produce high-fidelity computations for experimental validation using high-quality meteorological, turbine, and wake measurement data collected at the Department of Energy/Sandia National Laboratories Scaled Wind Farm Technology (SWiFT) facility located at Texas Tech University’s National Wind Institute. The wake analysis showed the simulated lidar model implemented in Nalu-Wind was successful at capturing wake profile trends observed in the experimental lidar data.

17 WIND ENERGY↗

DOE OSTI2021

Encoder–decoder neural network for solving the nonlinear Fokker–Planck–Landau collision operator in XGC

An encoder–decoder neural network has been used to examine the possibility for acceleration of a partial integro-differential equation, the Fokker–Planck–Landau collision operator. This is part of the governing equation in the massively parallel particle-in-cell code XGC, which is used to study turbulence in fusion energy devices. The neural network emphasizes physics-inspired learning, where it is taught to respect physical conservation constraints of the collision operator by including them in the training loss, along with the ℓ 2 loss. In particular, network architectures used for the computer vision task of semantic segmentation have been used for training. A penalization method is used to enforce the ‘soft’ constraints of the system and integrate error in the conservation properties into the loss function. During training, quantities representing the particle density, momentum and energy for all species of the system are calculated at each configuration vertex, mirroring the procedure in XGC. This simple training has produced a median relative loss, across configuration space, of the order of 10 –4 , which is low enough if the error is of random nature, but not if it is of drift nature in time steps. The run time for the current Picard iterative solver of the operator is O(n 2 ), where n is the number of plasma species. As the XGC1 code begins to attack problems including a larger number of species, the collision operator will become expensive computationally, making the neural network solver even more important, especially since its training only scales as O(n). Here, a wide enough range of collisionality has been considered in the training data to ensure the full domain of collision physics is captured. An advanced technique to decrease the losses further will be subject of a subsequent report. Eventual work will include expansion of the network to include multiple plasma species.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2022

Ab-Initio Investigation of Finite Size Effects in Rutile Titania Nanoparticles with Semilocal and Nonlocal Density Functionals

In this work, we employ hybrid and generalized gradient approximation (GGA) level density functional theory (DFT) calculations to investigate the convergence of surface properties and electronic gap of rutile titania nanoparticles with particle size. The surface energies and electronic gaps are calculated for cuboidal particles with minimum dimension ranging from 3.7 Angstrom (24 atoms) to 10.3 Angstrom (384 atoms) using a highly-parallel real-space DFT code to enable hybrid level DFT calculations of larger nanoparticles than are typically practical. We deconvolute the geometric and electronic finite size effects in surface energy, and evaluate the influence of defects on electronic gap and density of states (DOS). The electronic finite size effects in surface energy vanish when the minimum length scale of the nanoparticles becomes greater than 10 Angstrom. We show that this length scale is consistent with a computationally efficient numerical analysis of the characteristic length scale of electronic interactions. The surface energy of nanoparticles having minimum dimension beyond this characteristic length can be approximated using slab calculations that account for the geometric defects. In contrast, the finite size effects on the electronic gap and DOS is highly dependent on the shape and size of these particles. Furthermore, the DOS for cuboidal particles and more realistic particles constructed using the Wulff algorithm reveal that defect states within the electronic gap play a key role in determining the eigen value distribution of nanoparticles and the electronic gap does not converge to the bulk limit for the particle sizes investigated.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗