Search NASA⌕ Search

SEARCH · Search NASA

Results for “Parallel Fluid Dynamic Solver”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Investigating the Interaction Between Circulating Tumor Cells and Local Hydrodynamics via Experiment and Simulations

Introduction: The biological and mechanical properties of circulating tumor cells (CTCs) in combination with the hemodynamics affect the preference of metastatic sites in the vasculature. Despite the extensive literature on the effects of biological properties on cell adhesion, the effects of hydrodynamic forces on primary attachment remains an active area of research. Here, using simulations in conjunction with experimentation, we provide new insight into the interplay of CTCs dynamics and local hydrodynamics. Methods: A flow experiment of CTC attachment was performed within a bioprinted, double branching endothelialized vessel. Simulations of fluid flow and CTC transport in the reconstructed and idealized bifurcated vessel were respectively performed by HARVEY, our in-house massively parallel computational fluid dynamics solver. HARVEY is based on the lattice Boltzmann and finite element methods to model the fluid and cells dynamics. The immersed boundary method is employed for resolving the fluid–structure interaction. Results: CTC attachment was quantified experimentally at all regions of the complex vessel. The results demonstrate a clear preference for CTCs to attach at the branch points. To elucidate the effect of the vessel topology on the location of attachment, a fluid-only simulation was performed assessing the differences in the hydrodynamics along the vessel. CTC transport in idealized bifurcated vessels was subsequently studied to examine the effects of cell deformability on the local hydrodynamics patterns and, thus, the preference of attachment sites. Conclusions: The current work provides evidence on the correlation of the hydrodynamics forces arising from the vessel topology and CTC properties on the attachment regions.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Coupling hydrodynamics to a rigid-body motion solver for fluid-structure interaction [Slides]

Quinoa is a massively parallel computational fluid dynamics with multi-material and programmed burn capabilities developed from Programmatic and LDRD funds. Overset is a well-established method of using an “inset” mesh that communicates with a background mesh. Solutions transfer freely from one to another and operate as boundary conditions on the opposite mesh. Either mesh can move at specified velocity! This project couples the existing Mesh-to-mesh transfer with Quinoa and demonstrates use of resulting Overset solver in computational simulation of blast effects on a re-entry body for survivability assessments.

42 ENGINEERING↗
DOE OSTI2025

A Comparison of GPU-Accelerated Multiphase CFD Solvers on the Polaris Supercomputer: Part 1

This report is in support of the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program sponsored by the U.S. Department of Energy (USDOE). With INCITE-level resources, one project, titled BubblyFlow, was granted computational resources for the 2025 calendar year on the Polaris supercomputer at the Argonne Leadership Computing Facility (ALCF). The project aims to conduct simulations to understand the fundamental characteristics of turbulent bubbly flow phenomena in nature. Staff at the ALCF and Argonne’s Computational Science division, along with collaborators at the City College of New York and University of Illinois at Chicago, helped a summer student to assess the accuracy and performance of two high performance computing (HPC) codes. Both codes, ImExLBM and FluTAS, are fundamentally different in their mathematical and numerical modeling. However, both may be used to solve the same physical problem. The collaboration sought to better understand the differences between both codes in terms of accuracy and efficiency. This would ultimately help the BubblyFlow project better utilize resources and establish a knowledge-base of code capabilities in future simulation campaigns. We compare ImExLBM and FluTAS, two high-performance multiphase computational fluid dynamics (CFD) solvers, in terms of physical fidelity, time-to-solution, and parallel efficiency. We validate ImExLBM (Implicit-Explicit Lattice Boltzmann Method) against a canonical benchmark and assess it’s performance relative to FluTAS (Fluid Transport Accelerated Solver), a well-established open-source CFD code.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

kynema-sgf [SWR-20-85]

Kynema-SGF (formerly AMR-Wind), wherein SGF stands for structured-grid fluid dynamics, is a massively parallel, block-structured adaptive-mesh, incompressible flow solver. The codebase was initiated in 2019 from incflo. The solver is built on top of the AMReX library. AMReX library provides the mesh data structures, mesh adaptivity, as well as the linear solvers used for solving the governing equations. Kynema-SGF is actively developed and maintained by a dedicated multi-institutional team from Lawrence Berkeley National Laboratory, National Laboratory of the Rockies, and Sandia National Laboratories. The primary applications for Kynema-SGF are: performing large-eddy simulations (LES) of atmospheric boundary layer (ABL) flows, simulating wind farm turbine-wake interactions using actuator disk or actuator line models for turbines, and as a background solver when coupled with a near-body solver (e.g., Kynema-UGF) with overset methodology to perform blade-resolved simulations of multiple wind turbines within a wind farm. For offshore applications, the ability to model the air-sea interaction effects and its impact on the ABL characteristics is another focus for the code development effort. As with other codes in the Kynema ecosystem, Kynema-SGF shares the following objectives: *an open, well-documented implementation of the state-of-the-art computational models for modeling wind farm flow physics at various fidelities that are backed by a comprehensive verification and validation (V&V) process; *be capable of performing the highest-fidelity simulations of flow fields within wind farms; and *be able to leverage the high-performance leadership class computing facilities available at DOE national laboratories.

Ananthan, Shreyas↗
DOE OSTI2019

Reverse-Mode Algorithmic Differentiation of an OpenMP-Parallel Compressible Flow Solver

y Reverse-mode algorithmic differentiation (AD) is an established method for obtaining adjoint derivatives of computer simulation applications. In computational fluid dynamics (CFD), adjoint derivatives of a cost function output such as drag or lift with respect to design parameters such as surface coordinates or geometry control points are a key ingredient for shape optimization, uncertainty quantification and flow control. The computational cost of CFD applications and their derivatives makes it essential to use high-performance computing hardware efficiently, including multi- and many-core architectures. Nevertheless, OpenMP is not supported in most AD tools, and previously shown methods achieve poor scalability of the derivative code. We present the AD of an OpenMP-parallelized finite volume compressible flow solver for unstructured meshes. Our approach enables us to reuse the parallelization of the original code in the computation of adjoint derivatives. The method works by identifying code segments that can be differentiated in reverse-mode without changing their memory access pattern. The OpenMP parallelization is integrated into the derivative code during the build process in a way that is robust to modifications of the original code and independent of the OpenMP support of the differentiation tool. We show the scalability of our adjoint CFD solver on test cases ranging from thousands to millions of finite volume mesh cells on CPUs with up to 16 threads as well as on an Intel XeonPhi card with 236 threads. We demonstrate that our approach is more practical to implement for production-sized CFD codes and produces more efficient adjoint derivative code than previously shown AD methods.

Huckelheim, Jan↗
DOE OSTI2019

Asynchronous Navier-Stokes Solver on 3D Unstructured Grids for the Exascale Era

This project has developed multiple fluid dynamics solvers for complex 3D flows using fully asynchronous distributed-memory task-parallel algorithms on top of the Charm++ runtime system. The algorithms solve the Euler or Navier-Stokes equations of compressible flows using unstructured tetrahedron meshes with optional solution-adaptive mesh-, and polynomial-order refinement. We have demonstrated excellent strong scaling up 50K compute cores and the benefits of Charm++’s automatic load balancing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A numerical solver for investigating the space charge effect on the electric field in liquid argon time projection chambers

This paper reports the development of a numerical solver aimed to simulate the interaction between the space charge (i.e. ions) distribution and the electric field in liquid argon time projection chamber (LArTPC) detectors. The ion transport equation is solved by a time-accurate, cell-centered finite volume method and the electric potential equation by a continuous finite element method. The electric potential equation updates the electric field which provides the drift velocity to the ion transport equation. The ion transport equation updates the space charge density distribution which appears as the source term in the electric potential equation. The interaction between the space charge distribution and the electric field is numerically simulated within each physical time step. The convective velocity in the ion transport equation can include the background flow velocity in addition to the electric drift velocity. The numerical solver has been parallelized using the Message Passing Interface (MPI) library. Numerical tests show and verify the capability and accuracy of the current numerical solver. It is planned that the developed numerical solver, together with a Computational Fluid Dynamics (CFD) package which provides the flow velocity field, can be used to investigate the space charge effect on the electric field in large-scale particle detectors.

42 ENGINEERING↗
DOE OSTI2025

AMR-Wind: A Performance-Portable, High-Fidelity Flow Solver for Wind Farm Simulations

We present AMR-Wind, a verified and validated high-fidelity computational-fluid-dynamics code for wind farm flows. AMR-Wind is a block-structured, adaptive-mesh, incompressible-flow solver that enables predictive simulations of the atmospheric boundary layer and wind plants. It is a highly scalable code designed for parallel high-performance computing with a specific focus on performance portability for current and future computing architectures, including graphical processing units (GPUs). In this paper, we detail the governing equations, the numerical methods, and the turbine models. Establishing a foundation for the correctness of the code, we present the results of formal verification and validation. The verification studies, which include a novel actuator line test case, indicate that AMR-Wind is spatially and temporally second-order accurate. The validation studies demonstrate that the key physics capabilities implemented in the code, including actuator disk models, actuator line models, turbulence models, and large eddy simulation (LES) models for atmospheric boundary layers, perform well in comparison to reference data from established computational tools and theory. We conclude with a demonstration simulation of a 12-turbine wind farm operating in a turbulent atmospheric boundary layer, detailing computational performance and realistic wake interactions.

17 WIND ENERGY↗
DOE OSTI2020

Multiscale formulation of frictional contact mechanics at the pore scale

Direct numerical simulation (DNS) yields the highest fidelity predictions of mechanical deformation at the pore scale, but is prohibitively expensive for analyzing large or many samples. Discrete element methods (DEM) are an efficient alternative, but are limited to granular media and incapable of estimating or controlling prediction errors. We present a pore-level multiscale method (PLMM) that approximates DNS efficiently and with controllable accuracy. We focus on the linear elastic response of a consolidated geologic porous medium with arbitrary microstructure, heterogeneous mineralogy, containing cracks or defects. PLMM decomposes the solid phase into non-overlapping subdomains, on which local basis functions are constructed. The bases are then coupled with a global interface problem that accounts for slip or stick contact conditions between the subdomains. PLMM produces an initial, but accurate, approximation to DNS that can be iteratively improved. It is amenable to parallelism and allows for different mesh, models, and physics in each subdomain. An algebraic interpretation of PLMM as a preconditioner is also presented to allow non-intrusive implementation into existing solvers. Lastly, this work extends previous developments of PLMM in fluid dynamics to solid mechanics and enables future extensions towards modeling coupled flow and mechanics problems.

42 ENGINEERING↗
DOE OSTI2019

Nalu-Wind and OpenFAST: A high-fidelity modeling and simulation environment for wind energy. Milestone ECP-Q2-FY19

The goal of the ExaWind project is to enable predictive simulations of wind farms comprised of many megawatt-scale turbines situated in complex terrain. Predictive simulations will require computational fluid dynamics (CFD) simulations for which the mesh resolves the geometry of the turbines and captures the rotation and large deflections of blades. Whereas such simulations for a single turbine are arguably petascale class, multi-turbine wind farm simulations will require exascale-class resources. The primary physics codes in the ExaWind project are Nalu-Wind, which is an unstructured-grid solver for the acoustically incompressible Navier-Stokes equations, and OpenFAST, which is a whole-turbine simulation code. The Nalu-Wind model consists of the mass-continuity Poisson-type equation for pressure and a momentum equation for the velocity. For such modeling approaches, simulation times are dominated by linear-system setup and solution for the continuity and momentum systems. For the ExaWind challenge problem, the moving meshes greatly affect overall solver costs as reinitialization of matrices and recomputation of preconditioners is required at every time step. This milestone represents the culmination of several parallel development activities towards the goal of establishing a full-physics simulation capability for modeling wind turbines operating in turbulent atmospheric inflow conditions. The demonstration simulation performed in this milestone is the first step towards the "ground truth" simulation and includes the following components: neutral atmospheric boundary layer inflow conditions generated using a precursor simulation, a hybrid RANS/LES simulation of the wall-resolved turbine geometry, hybridization of the turbulence equations using a blending function approach to transition from the atmospheric scales to the blade boundary layer scales near the turbine, fluid-structure interaction (FSI) that accounts for the complete set of blade deformations (bending, twisting and pitch motion, yaw and tower displacements) by coupling to a comprehensive turbine dynamics code (OpenFAST). The use of overset mesh methodology for the simulations in this milestone presents a significant deviation from the previous efforts where a sliding mesh approach was employed to model the rotation of the turbine blades. The choice of overset meshes was motivated by the need to handle arbitrarily large deformations of the blade and to allow for blade pitching in the presence of a controller and the ease of mesh generation compared to the sliding mesh approach. FSI and the new timestep algorithm used in the simulations were developed in partnership with the A2e High-Fidelity Modeling project. The individual physics components were verified and validated (V%V) through extensive code-to-code comparisons and with experiments where possible. The detailed V&V efforts provide confidence in the final simulation where these physics models were combined together even though no detailed experimental data is available to perform validation of the final configuration. Taken together, this milestone successfully demonstrated the most advanced simulation to date that has been performed with Nalu-Wind.

17 WIND ENERGY↗
DOE OSTI2020

The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers

The design and implementation of a new framework for adaptive mesh refinement calculations are described. It is intended primarily for applications in astrophysical fluid dynamics, but its flexible and modular design enables its use for a wide variety of physics. The framework works with both uniform and nonuniform grids in Cartesian and curvilinear coordinate systems. It adopts a dynamic execution model based on a simple design called a "task list" that improves parallel performance by overlapping communication and computation, simplifies the inclusion of a diverse range of physics, and even enables multiphysics models involving different physics in different regions of the calculation. We describe physics modules implemented in this framework for both nonrelativistic and relativistic magnetohydrodynamics (MHD). These modules adopt mature and robust algorithms originally developed for the Athena MHD code and incorporate new extensions: support for curvilinear coordinates, higher-order time integrators, more realistic physics such as a general equation of state, and diffusion terms that can be integrated with super-time-stepping algorithms. The modules show excellent performance and scaling, with well over 80% parallel efficiency on over half a million threads. The source code has been made publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Preconditioning a Newton-Krylov solver for all-speed melt pool flow physics

n this paper, we introduce a multigrid block-based preconditioner for solving linear systems arising from a Discontinuous Galerkin discretization of the all-speed Navier-Stokes equations with phase change. The equations are discretized in conservative form with a reconstructed Discontinuous Galerkin (rDG) method and integrated with fully-implicit time discretization schemes. To robustly converge the numerically stiff systems, we use the Newton-Krylov framework with a primitive-variable formulation (pressure, velocity, and temperature), which is better conditioned than the conservative-variable form at low-Mach number. In the limit of large acoustic CFL number and viscous Fourier number, there is a strong coupling between the velocity-pressure system and the linear systems become non-diagonally dominant. To effectively solve these ill-conditioned systems, an approximate block factorization preconditioner is developed, which uses the Schur complement to reduce a 3 x 3 block system into a sequence of two 2 x 2 block systems: velocity-pressure,v P , and velocity-temperature, v T . We compare the performance of the v P -v T Schur complement preconditioner to classic preconditioning strategies: monolithic algebraic multigrid (AMG), element-block SOR, and primitive variable block Gauss-Seidel. The performance of the preconditioned solver is investigated in the limit of large CFL and Fourier numbers for low-Mach lid-driven cavity flow, Rayleigh-Bénard melt convection, compressible internally heated convection, and 3D laser-induced melt pool flow. Here, numerical results demonstrate that the v P -v T Schur complement preconditioned solver scales well both algorithmically and in parallel, and is robust for highly ill-conditioned systems, for all tested rDG discretization schemes (up to 4th-order).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Evaluation of intracoronary hemodynamics identifies perturbations in vorticity

Coronary artery disease (CAD) is highly prevalent and associated with adverse events. Challenges have emerged in the treatment of intermediate coronary artery stenoses. These lesions are often interrogated with fractional flow reserve (FFR) testing to determine if a stenosis is likely to be causative for ischemia in a cardiac territory. This invasive test requires insertion of a pressure wire into a coronary vessel. Recently computational fluid dynamics (CFD) has been used to noninvasively assess fractional flow reserve in vessels reconstructed from medical imaging data. However, many of these simulations are unable to provide additional information about intravascular hemodynamics, including velocity, endothelial shear stress (ESS), and vorticity. We hypothesized that vorticity, which has demonstrated utility in the assessment of ventricular and aortic diseases, would also be an important hemodynamic factor in CAD. Three-dimensional (3D), patient-specific coronary artery geometries that included all vessels >1 mm in diameter were created from angiography data obtained from 10 patients who underwent diagnostic angiography and FFR testing (n = 9). A massively parallel CFD solver (HARVEY) was used to calculate coronary hemodynamic parameters including pressure, velocity, ESS, and vorticity. These simulations were validated by comparing velocity flow fields from simulation to both velocities derived from in vitro particle image velocimetry and to invasively acquired pressure wire-based data from clinical testing. There was strong agreement between findings from CFD simulations and particle image velocimetry experimental testing (p < 0.01). CFD-FFR was also highly correlated with invasively measured FFR (ρ = 0.77, p = 0.01) with an average error of 5.9 ± 0.1%. CFD-FFR also had a strong inverse correlation with the vorticity (ρ = -0.86, p = 0.001). Simulations to determine the effect of the coronary stenosis on intravascular hemodynamics demonstrated significant differences in velocity and vorticity (both p < 0.05). Further evaluation of an angiographically normal appearing non-FFR coronary vessel in patients with CAD also demonstrated differences in vorticity when compared with FFR vessels (p < 0.05). The use of highly accurate 3D CFD-derived intravascular hemodynamics provides additional information beyond pressure measurements that can be used to calculate FFR. Vorticity is one parameter that is modified by a coronary stenosis and appears to be abnormal in angiographically normal vessels in patients with CAD, highlighting a possible use-case in preventative screening for early coronary disease.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

MARBLES (Multi-scale Adaptively Refined Boltzmann LatticE Solver) [SWR-23-37]

MARBLES (Multi-scale Adaptively Refined Boltzmann LatticE Solver) is an open-source computational fluid dynamics package powered by the lattice Boltzmann equations and built on AMReX. In the lattice Boltzmann method, local collisions between meso-scale fictitious particles drive the governing equations which enables MARBLES to easily simulate flow around complex and/or moving geometry without the generation of a body-conforming mesh. Using AMReX data structures and operations ensures a high level of computational performance and parallel scaling on heterogenous architectures while also naturally supporting locally enhanced grid resolution and fidelity through automatic mesh refinement. New domains and problem definitions are easily specified through an input file with examples and guidance on all options and variables provided in the MARBLES documentation.

Henry de Frahan, Marc↗
DOE OSTI2020

An Adaptive-Mesh-Refinement Based Computational Tool for Simulating Catalysis at Mesoscale

In this work, we present a computational tool for mesoscale applications using open-source exascale- computing compatible adaptive-mesh-refinement (AMR) library, AMReX [2]. AMReX is software library that enables development of application solvers with block-structured Cartesian AMR. Our tool has capabilities to include realistic geometry representation, chemical species transport, reactions and thermodynamics that are critical for capturing mesoscale physics. A significant achievement is the ability of our solver to automatically import electron microscopy data in the form of a stereolithography (STL) or pixelated file format (mrc, tiff) without undergoing the tedious task of unstructured mesh generation. This feature allows for rapid simulation of catalyst particles with complex morphologies using an immersed-boundary formulation. The use of AMR allows for higher resolutions at catalyst surface interfaces, which in turn provides an accurate description of surface reactions and transport. Our solver uses a hybrid distributed and shared memory parallelism (OpenMP/GPU-based) with which strong scaling up to 10,000 processors for realistic catalyst particle simulations have been demonstrated.

BIOMASS FUELS,MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Multigrid treatment of implicit continuum diffusion

Implicit treatment of diffusive terms of various differential orders common in continuum mechanics modeling, such as computational fluid dynamics, is investigated with spectral and multigrid algorithms in non-periodic 2D domains. In doubly periodic time dependent problems these terms are handled efficiently by spectral methods, but in non-periodic systems solved with distributed memory parallel computing and 2D domain decomposition, this efficiency is lost for a large number of processors. Here, we built and present here a multigrid algorithm for these types of problems that outperforms a spectral solution employing the highly optimized FFTW library. This solver is suitable for high performance computing and may be able to efficiently treat implicit diffusion of arbitrary order by introducing auxiliary equations of lower order. We test these solvers for fourth and sixth order diffusion with harmonic test functions as well as turbulent 2D magnetohydrodynamic simulations. It is also shown that an anisotropic operator without mixed-derivative terms improves model accuracy and speed, and we examine the impact that the various diffusion operators have on the energy, the enstrophy, and the qualitative aspect of a simulation.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Applying Time-Parallelization to Turbulent Flows

Parallelization of the temporal domain is explored for the solution of turbulent flows. Multigrid reduction-in-time (MGRIT) is used to advance the large-scale fluid dynamics in time sequentially on the coarsest space-time grid but propagate the information in time parallel on all other levels. The goal of this process is to accurately and efficiently resolve the coarse-scale turbulence structure and use that to drive the fine-scales of the turbulent flow. The extra forcing from nonlinear multigrid facilitates the coupling and interaction between fine and coarse scales, through which the multiscale nonlinear physics is properly captured. Adaptive mesh refinement is employed to finely resolve only the regions with strong gradients, which provides further computational efficiency. The underlying computational fluid dynamics solver is a fourth-order finite-volume scheme with the standard 4-stage Runge-Kutta method. An advanced approach is devised and implemented to enable MGRIT to solve highly turbulent flows successfully. Furthermore, the method is applied to solve a Taylor-Green vortex problem and a doubleshear-layer turbulent mixing flow. Results are promising, validating that MGRIT with the filtering approach has the potential to efficiently solve general turbulent flows.

Computational Fluid Dynamics↗