SEARCH · Search NASA
Results for “massively parallel simulations”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Developing Multiphysics, Integrated, High-Fidelity, Massively Parallel Computational Capabilities for Fusion Applications Using MOOSE
As the need for fusion as a clean, sustainable, and abundant energy source grows internationally, so does the need for multiphysics, computational tools to model, study, and predict the complex interactions between plasma, materials, and engineering processes. These tools have a crucial role to play in solving scientific and engineering challenges and accelerating fusion energy deployment. To address these needs, modeling capabilities should enable massively parallel, multiphysics, fully integrated high-fidelity simulations of fusion systems. Additional attributes, such as being open source and modular while maintaining high software quality assurance standards will maximize impact by ensuring accessibility for all and wide acceptance, rapid expansion and development, as well as reliability, efficiency, and robustness. In this paper, we describe how the Multiphysics Object-Oriented Simulation Environment (MOOSE) framework, which has a track record of success in the fission space thanks to the attributes listed above, can be leveraged in the fusion energy field. We highlight key successes of the MOOSE application in the fission space and describe how MOOSE has been and is being applied to fusion applications in the United States---e.g., Tritium Migration Analysis Program, version 8 (TMAP8), MOOSE Fusion Module, Fusion ENergy Integrated multiphys-X (FENIX)---and the United Kingdom---e.g., AURORA, Achlys, Apollo. These efforts aim to establish a suite of tools that can be further extended to accelerate fusion energy deployment.
Massively parallel axisymmetric fluid model for streamer discharges
A highly parallelizable fluid plasma simulation tool based upon the first-order drift-diffusion equations is discussed. Atmospheric pressure plasmas have densities and gradients that require small element sizes in order to accurately simulate the plasm resulting in computational meshes on the order of millions to tens of millions of elements for realistic size plasma reactors. To enable simulations of this nature, parallel computing is required and must be optimized for the particular problem. Here, a finite-volume, electrostatic drift-diffusion implementation for low-temperature plasma is discussed. The implementation is built upon the Message Passing Interface (MPI) library in C++ using Object Oriented Programming. The underlying numerical method is outlined in detail and benchmarked against simple streamer formation from other streamer codes. Electron densities, electric field, and propagation speeds are compared with the reference case and show good agreement. Convergence studies are also performed showing a minimal space step of approximately 4 μm required to reduce relative error to below 1% during early streamer simulation times and even finer space steps are required for longer times. Additionally, strong and weak scaling of the implementation are studied and demonstrate the excellent performance behavior of the implementation up to 100 million elements on 1024 processors. Lastly, different advection schemes are compared for the simple streamer problem to analyze the influence of numerical diffusion on the resulting quantities of interest.
Interface capturing simulations of bubble population effects in PWR subchannels
As the computational power of high-performance computing (HPC) facilities grows, so too does the feasibility of using first principle based simulation to study turbulent two-phase flows within complex pressurized water reactor (PWR) geometries. Direct numerical simulation (DNS), integrated with an interface capturing method, allows for the collection of high-fidelity numerical data using advanced analysis techniques. The research presented here employs the massively parallel, finite-element based, unstructured mesh code, PHASTA, to simulate a set of two-phase bubbly flows through PWR subchannel geometries including auxiliary structures (spacer grids and mixing vanes). The main objective of the presented work is to analyze bubble dynamics and turbulence interactions at varying bubble concentrations to support the development of advanced two-phase flow closure models. Turbulent two-phase flows in PWR subchannels were simulated at hydraulic Reynolds numbers of 81,000 with bubble concentrations of 3%–15% by gas volume fraction (768–3928 resolved bubbles, respectively) and compared against a 1% void fraction case (262 bubbles) that had been previously simulated. The finite element mesh utilized for the study at higher bubble concentrations was composed of 1.55 billion elements, compared to the previous study which employed 1.11 billion elements, ensuring all turbulence scales and individual bubbles within the flow are fully resolved. For each case, the resolved initial bubble size was 0.65 mm in diameter (resolved with 25 grid points across the diameter). The simulations were analyzed to find flow features such as the mean velocity profile, bubble relative velocity and the effect of the bubbles on the turbulent conditions.
MFIX DEM Enhancement for Industry-Relevant Flows (Final Report)
The overall goal of this two-phase project is to implement performance improvements of the Multiphase Flow with Interphase Exchanges (MFIX) Discrete Element Model (DEM) code that enable a transformative shift for industrial use. Prior to this effort, the largest simulations performed using MFIX are O(10 7 ) particles. This falls short of the O(10 9 ) particle simulations that must be completed on a timescale of days or weeks (vs. months or years) to enable simulations with physically-relevant domain sizes to be incorporated into industrial design cycles within five years. This was accomplished by tailoring best-in-class practices to bear on the unique challenges posed by the MFIX-DEM algorithm and code base. Scientific simulations (e.g., in cosmology, turbulent combustion) routinely use massively parallel computing to update far more particles in short wall clock times. Results from Phase 1 (1.5 years in duration) indicated significant gains in speed were possible for a wide range of benchmark cases. Moreover, a survey sent to >35 companies indicates that the timing is ideal for such an enhanced tool, with >80% of the respondents indicating that DEM is already value-added or will be within the next 5 years, and >70% of the respondents indicating that improved speed is the top computational priority. In Phase 2 (3.5 years in duration), the two major barriers that hinder industry from effectively using multiphase Computational Fluid Dynamics (CFD) to cut costs and improve performance, namely computational overhead and confidence in predictions, continued to be addressed. Regarding the former, the results from Phase 1 to guide the effort, with enhancements focused on an improved time-stepping algorithm and particle sorting. Four target problems of 1 billion particles each and increasing complexity were identified: homogeneous cooling, tumbler with continuous particle size distribution, discharge from a rectangular hopper and a cylindrical riser. Each of these were successfully simulated for relevant time scales (on order of seconds) using less than 24 hours of wall clock time. These represent the first 1-billion particle DEM simulations performed with MFIX, namely using the MFIX-Exa code. This code is currently under development at NETL in collaboration with Lawrence Berkeley National Laboratory. Regarding the second barrier on predictive uncertainty, experiments from Phase 1 (interacting nozzles - hydrodynamics only) and Phase 2 (very small-scale segregation experiments) were used to demonstrate the ability of two simplified approaches to uncertainty quantification (UQ). By limiting the number of particles, UQ based on the simplified treatment was compared to standard UQ, which was shown to have much higher computational demands. Experiments were also performed on a pilot-scale stripper unit to provide validation data for future CFD-DEM simulations and UQ.
Porting Classical Approaches for Quantum Simulations to Quantum Computers
Simulating quantum many-body systems is one of the most promising problems in which we might anticipate that quantum computers should show quantum advantage. Unfortunately, there is still a gap between this promise and actual practice. New quantum algorithms need to be developed and the current quantum algorithms have various difficulties - e.g efficient state preparation - which must be overcome and improved upon. In many cases, classical approaches need to be ported over to quantum devices. In this project we have developed a suite of new quantum algorithms which makes progress in this regard. We developed a new optimization scheme for variational quantum eigensolvers, UBOS, which mitigates problems with local minimas and barren plateaus while improving convergence to the ground state by an order of magnitude. We developed a new way to utilize qubitization to find ground states of nearly frustration-free Hamiltonians faster than all previous methods. We developed a series of state preparation techniques which helps initialize parameterized quantum circuits into reasonable starting points on which quantum algorithms are then applied. In addition to the development of novel algorithms, it is critical to have classical simulation techniques for approximately simulating quantum circuits which can be used to benchmark and understand quantum algorithms. Toward that end, we developed a novel POVM formalism to simulate quantum circuits as well as exemplify the massive parallelization of tensor network methodologies. Finally, we developed physical understanding of entanglement phase transitions such as many-body localization and random tensor networks.
SPARC-X: Quantum simulations at extreme scale - reactive dynamics from first principles
We have developed the massively parallel electronic structure code SPARC-X: a computational framework for performing Kohn-Sham Density Functional Theory (DFT) calculations that can scale linearly with the number of atoms in the system, while being able to leverage petascale and emerging exascale parallel computers to study chemical phenomena at unprecedented length and time scales. SPARC-X exploits a recent breakthrough in electronic structure methodologies: systematically improvable, strictly local, orthonormal, discontinuous real-space bases that efficiently and systematically capture the local chemistry of the system. With further adaptation using new machine-learning techniques and the use of the massively parallel Spectral Quadrature (SQ) electronic structure method, the algorithmic complexity and prefactor associated with DFT calculations involving semilocal as well as hybrid functionals are dramatically reduced. Using petascale computational resources, SPARC-X enables quantum mechanical simulations at length and time scales previously accessible only by empirical approaches, e.g., 1,000,000 atoms for a few picoseconds using semilocal functionals or 1,000 atoms for a few picoseconds using hybrid functionals. Using exascale resources, the sizes and times targeted are two orders of magnitude larger. Such a capability has applications in a wide variety of chemical sciences, including reactive interfaces where large length- and/or long time-scales are needed and traditional force fields fail. This is particularly important in dynamic catalysis, where bond breaking and formation must be understood in detail. We developed, tested, and employed the SPARC-X framework to understand the photocatalytic properties of TiO 2 nanoparticles, revealing finite size effects that cannot be captured with standard model systems or functionals. This integrated development and application strategy ensures that SPARC-X remains a robust, efficient, and scalable software package for quantum simulations on current petascale and emerging exascale computing resources.
BISON Robustness and Performance Improvements
BISON is a modern finite-element based nuclear fuel performance code that has been under development at the Idaho National Laboratory (USA) since 2009 [1]. The code is applicable to both steady and transient fuel behavior and can be used to analyze 1D (spherically symmetric), 2D (axisymmetric and generalized plane strain) or 3D geometries. BISON is the fuel performance code used within CASL for LWR fuel under both normal operating and accident conditions. BISON is built using the INL Multiphysics ObjectOriented Simulation Environment, or MOOSE [2, 3]. MOOSE is a massively parallel, finite element-based framework to solve systems of coupled non-linear partial differential equations using the Jacobian-Free Newton Krylov (JFNK) method [4]. This enables investigation of computationally large problems, for example a full stack of discrete pellets in a LWR fuel rod, or every rod in a full reactor core. MOOSE supports the use of complex two and three-dimensional meshes and uses implicit time integration, important for the widely varied time scale in nuclear fuel simulation. An object-oriented architecture is employed which greatly minimizes the programming effort required to add new material and behavioral models. The flexibility of the implicit and fully coupled multiphysics approach comes with a need for constructing suitable approximations for the Jacobian matrix of the coupled system used for either preconditioning a Krylov solve or in a direct Newton solve. Preconditioning options for Bison problems need to be revisited with new preconditioning methods becoming available.
CP2K: An Electronic Structure and Molecular Dynamics Software Package - Quickstep: Efficient and Accurate Electronic Structure Calculations
CP2K is an open source electronic structure and molecular dynamics software package to perform atomistic simulations of solid-state, liquid, molecular and biological systems. It is especially aimed at massively-parallel and linear-scaling electronic structure methods and state-of-the-art ab-initio molecular dynamics simulations. Excellent performance for electronic structure calculations is achieved using novel algorithms implemented for modern high-performance computing systems. This review revisits the main capabilities of CP2K to perform efficient and accurate electronic structure simulations. The emphasis is put on density functional theory and multiple post-Hartree-Fock methods using the Gaussian and plane wave approach and its augmented all-electron extension. TDK has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 716142). VRR has been supported by the Swiss National Science Foundation in the form of Ambizione grant No. PZ00P2 174227 and RZK by the Natural Sciences and Engineering Research Council of Canada (NSERC) through Discovery Grants (RGPIN-2016-0505). GKS and CJM are supported by the US Department of Energy, Office of Science, Office of Basic Energy Sciences, Division of Chemical Sciences, Geosciences, and Biosciences. UK based work was funded under the embedded CSE programme of the ARCHER UK National Supercomputing Service (http://www.archer.ac.uk), grants eCSE03-011, eCSE06-6, eCSE08-9, eCSE13-17 and the EPSRC (EP/P022235/1) grant “Surface and Interface Toolkit for the Materials Chemistry Community". Computational resources were provided by the Swiss National Supercomputing Centre (CSCS) and Compute Canada. The generous allocation of computing time on the FPGA-based supercomputer “Noctua" at PC2 is kindly acknowledged.
Massively Parallel Bayesian Model Calibration and Uncertainty Quantification with Applications to Nuclear Fuels and Materials
The U.S. Department of Energy (DOE)’s Nuclear Energy Advanced Modeling and Simulation (NEAMS) program aims to develop predictive capabilities by applying computational methods to the analysis and design of advanced reactor and fuel cycle systems. This program has been providing engineering-scale support for the development of BISON, a high-fidelity and high-resolution fuel performance tool. Fuel behavior in a nuclear reactor is governed by a complex network of mechanisms interacting with various other physics aspects in the reactor system. Any model developed to represent the fuel behavior will likely be idealized resulting in uncertainties in their predictions compared to the observed data. As such, this report was motivated by the need to identify the sources of uncertainties and quantify and propagate them through the fuel model outputs. Such quantification of uncertainties will establish a level of model trustworthiness, identify approaches to improve the model trustworthiness, and even guide optimal experiment design for maximal information gain. To accomplish the uncertainty quantification for computational models, this report has relied on the Bayesian framework which provides probabilistic treatment of models their inputs and outputs. The current state-of-the-art on performing Bayesian Uncertainty Quantification (UQ) for nuclear engineering models using High Performance Computing (HPC) resources have been reviewed. Implementation of capabilities for massively parallel Bayesian UQ in Multiphysics Object-Oriented Simulation Environment (MOOSE) is discussed. Several verification cases are discussed to verify the accuracy of the quantified uncertainties using the developed computational capabilities in MOOSE. Then, the problem of quantifying the uncertainties in TRI-Structural isOtropic (TRISO) fuel silver release is addressed. For the first time, the uncertainties arising from the TRISO Fission Gas Release (FGR) model due to model inadequacy and experimental noise are quantified. Also, the Bayesian capabilities are applied to the calibration of the MATPRO creep model, a widely used model in several fuel assessment cases. The impact of the prediction uncertainties in the MATPRO model on the fuel cladding behavior as part of the TRIBULATION assessment case (which is an integral effects case) is investigated. This report concludes with a discussion on the future work for the UQ for computational models.
Hybrid RANS-LES of the Atmospheric Boundary Layer for Wind Farm Simulations: Preprint
Wind farm simulations often do not accurately represent wake-atmospheric boundary layer (ABL) interactions, blade boundary layer (BL) dynamics, and turbine-turbine interactions. In this work, we use Active Model Split (AMS), a new hybrid Reynolds-Averaged Navier Stokes (RANS)-large eddy simulation (LES) model, which is well suited to capture these effects because the model can (i) accurately simulate the ABL with the Coriolis effect, (ii) is accurate in adverse pressure gradients such as those near wind turbine blades, and (iii) has sufficiently low computational cost to simulate multiple turbines while resolving the blade BL. For simplicity and consistency we develop AMS to be used throughout the domain rather than in a zonal method. We implement our work in the massively parallel flow solver, Nalu-Wind, so that our model can access the compute resources needed for blade-resolved simulations of multiple wind turbines. To accomplish these aims, we modify the baseline AMS by changing the RANS contribution to SST k - omega with a length scale limiter, adding the Coriolis effect, and developing an appropriate wall treatment. We show that AMS of the ABL with the Coriolis effect matches LES reference results better than those obtained with RANS. We describe our plans to add buoyancy effects and wind turbines to our AMS simulations.
Automated calculation and convergence of defect transport tensors
Defect diffusion is a key process in materials science and catalysis, but as migration mechanisms are often too complex to enumerate a priori, calculation of transport tensors typically have no measure of convergence and require significant end-user intervention. These two bottlenecks prevent high-throughput implementations essential to propagate model-form uncertainty from interatomic interactions to predictive simulations. In order to address these issues, we extend a massively parallel accelerated sampling scheme, autonomously controlled by Bayesian estimators of statewide sampling completeness, to build atomistic kinetic Monte Carlo models on a state-space irreducible under exchange and space group symmetries. Focusing on isolated defects, we derive analytic expressions for drift and diffusion coefficients, providing a convergence metric by calculating the Kullback–Leibler divergence across the ensemble of diffusion processes consistent with the sampling uncertainty. The autonomy and efficacy of the method is demonstrated on surface trimers in tungsten and Hexa-interstitials in magnesium oxide, both of which exhibit complex, correlated migration mechanisms.
Development of PFLOTRAN Transport Capability for Use in the Waste Isolation Pilot Plant Performance Assessment - 20545
Waste Isolation Pilot Plant (WIPP) performance assessment (PA) calculations estimate the probability of radionuclide release from the repository to the land surface and across the land withdrawal boundary for a regulatory period of 10,000 years after facility closure. Simulations of flow and transport in the repository and the surrounding Salado Formation are foundational to the PA. Because proposed additional waste emplacement panels would result in an asymmetric repository layout, the US Department of Energy (DOE) is preparing to transition to use of a three-dimensional (3-D) model domain for simulation of flow and transport instead of the two-dimensional (2-D) flared grid domain currently used. DOE has charged Sandia National Laboratories with developing the capability necessary to simulate processes affecting flow and transport in the WIPP in PFLOTRAN, an open-source massively parallel multi-phase flow and reactive transport code. The new flow and transport capabilities developed in PFLOTRAN incorporate WIPP-specific process models and will replace the 2-D simulators (BRAGFLO and NUTS) that are currently utilized for Salado flow and transport calculations in WIPP PA. The focus of this paper is on the development of a new Nuclear Waste Transport (NWT) mode in PFLOTRAN that has all of the capabilities necessary for Salado transport simulations, including the ability to handle complete dry-out (100% gas saturation) of arbitrary cells in the model domain, radionuclide mass conservation at step changes in porosity associated with borehole intrusion, and the ability to calculate fluxes on a flared grid. The new PFLOTRAN transport capability and a suite of verification tests were designed around a list of functional requirements for WIPP PA calculations. (authors)
MrHyDE v.1.0
SAND2024-01324O MrHyDE, which stands for Multi-resolution Hybridized Differential Equations, is a general-purpose C++ package for the solution of coupled multiphysics and multiscale systems on massively parallel computing systems. MrHyDE is designed to enable moving beyond forward simulation for multiscale applications which includes optimization, control, uncertainty quantification, and stochastic inversion. The framework provides interfaces to several packages within the Trilinos framework and leverages automatic differentiation to enable adjoint capabilities for large-scale, gradient-based optimization. MrHyDE provides automated multiscale capabilities through a subgrid model interface and multiscale Dirichlet-to-Neumann maps. For extreme-scale applications, MrHyDE provides in situ data-compression algorithms to reduce memory requirements while maintaining performance. MrHyDE is a general-purpose, computational framework for the solution of multiscale and multiphysics applications. It uses a combination of structure-preserving, physics-compatible discretizations, fully implicit methods, multi-resolution schemes, or fully explicit methods. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
A fast particle-based approach for calibrating a 3-D model of the Antarctic ice sheet
We consider the scientifically challenging and policy-relevant task of understanding the past and projecting the future dynamics of the Antarctic ice sheet. The Antarctic ice sheet has shown a highly nonlinear threshold response to past climate forcings. Triggering such a threshold response through anthropogenic greenhouse gas emissions would drive drastic and potentially fast sea level rise with important implications for coastal flood risks. Previous studies have combined information from ice sheet models and observations to calibrate model parameters. These studies have broken important new ground but have either adopted simple ice sheet models or have limited the number of parameters to allow for the use of more complex models. These limitations are largely due to the computational challenges posed by calibration as models become more computationally intensive or when the number of parameters increases. Here, we propose a method to alleviate this problem: a fast sequential Monte Carlo method that takes advantage of the massive parallelization afforded by modern high-performance computing systems. We use simulated examples to demonstrate how our sample-based approach provides accurate approximations to the posterior distributions of the calibrated parameters. The drastic reduction in computational times enables us to provide new insights into important scientific questions, for example, the impact of Pliocene era data and prior parameter information on sea level projections. These studies would be computationally prohibitive with other computational approaches for calibration such as Markov chain Monte Carlo or emulation-based methods. We also find considerable differences in the distributions of sea level projections when we account for a larger number of uncertain parameters. For example, based on the same ice sheet model and data set, the 99th percentile of the Antarctic ice sheet contribution to sea level rise in 2300 increases from 6.5 m to 13.1 m when we increase the number of calibrated parameters from three to 11. With previous calibration methods, it would be challenging to go beyond five parameters. Here, this work provides an important next step toward improving the uncertainty quantification of complex, computationally intensive and decision-relevant models.
Protein Molecular Dynamics Simulations with Approximate QM: What Can We Learn?
Classical force fields are essential for computer simulations of proteins and are typically parameterized to reproduce secondary and tertiary structure of isolated proteins. However, while protein-protein interactions are ubiquitous in nature, they are not considered in parameterization efforts and are far less understood than isolated proteins. A better characterization of intermolecular interactions is widely recognized as a key to revolutionizing drug and therapeutic developments with high-throughput computational screening. Urgently needed is a critical assessment of the performance of modern protein force fields against first-principles electronic structure methods and experiments. In a daring step toward this goal, we here describe a comparison of peptide folding dynamics as predicted by a molecular mechanics force field on the one hand and by an approximate electronic structure quantum mechanical (QM) method based on density-functional tight-binding (DFTB) on the other. We further compare the dynamics from straightforward DFTB simulations with a near-linear scaling version of DFTB for massively parallel computation based on the fragment molecular orbital (FMO-DFTB) method. We illustrate differences between the phenomenology of the folding dynamics from these three methods for a small model peptide, as well as charge polarization and dynamic fluctuations, point out possible correlations and implications for force field developers, and discuss the lessons learned that might become applicable to future predictive high-throughput computer screening for personalized neoantigen cancer therapy.
Exascale models of stellar explosions: Quintessential multi-physics simulation
The ExaStar project aims to deliver an efficient, versatile, and portable software ecosystem for multi-physics astrophysics simulations run on exascale machines. The code suite is a component-based multi-physics toolkit, built on the capabilities of current simulation codes (in particular Flash-X and Castro), and based on the massively parallel adaptive mesh refinement framework AMReX. It includes modules for hydrodynamics, advanced radiation transport, thermonuclear kinetics, and nuclear microphysics. The code will reach exascale efficiency by building upon current multi- and many-core packages integrated into an orchestration system that uses a combination of configuration tools, code translators, and a domain-specific asynchronous runtime to manage performance across a range of platform architectures. The target science includes multi-physics simulations of astrophysical explosions (such as supernovae and neutron star mergers) to understand the cosmic origin of the elements and the fundamental physics of matter and neutrinos under extreme conditions.
Accelerating massively parallel hemodynamic models of coarctation of the aorta using neural networks
Comorbidities such as anemia or hypertension and physiological factors related to exertion can influence a patient’s hemodynamics and increase the severity of many cardiovascular diseases. Observing and quantifying associations between these factors and hemodynamics can be difficult due to the multitude of co-existing conditions and blood flow parameters in real patient data. Machine learning-driven, physics-based simulations provide a means to understand how potentially correlated conditions may affect a particular patient. Here, we use a combination of machine learning and massively parallel computing to predict the effects of physiological factors on hemodynamics in patients with coarctation of the aorta. We first validated blood flow simulations against in vitro measurements in 3D-printed phantoms representing the patient’s vasculature. We then investigated the effects of varying the degree of stenosis, blood flow rate, and viscosity on two diagnostic metrics – pressure gradient across the stenosis (ΔP) and wall shear stress (WSS) - by performing the largest simulation study to date of coarctation of the aorta (over 70 million compute hours). Using machine learning models trained on data from the simulations and validated on two independent datasets, we developed a framework to identify the minimal training set required to build a predictive model on a per-patient basis. We then used this model to accurately predict ΔP (mean absolute error within 1.18 mmHg) and WSS (mean absolute error within 0.99 Pa) for patients with this disease.