Search NASA⌕ Search

SEARCH · Search NASA

Results for “Parallel Solvers”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Hybrid RANS-LES of the Atmospheric Boundary Layer for Wind Farm Simulations: Preprint

Wind farm simulations often do not accurately represent wake-atmospheric boundary layer (ABL) interactions, blade boundary layer (BL) dynamics, and turbine-turbine interactions. In this work, we use Active Model Split (AMS), a new hybrid Reynolds-Averaged Navier Stokes (RANS)-large eddy simulation (LES) model, which is well suited to capture these effects because the model can (i) accurately simulate the ABL with the Coriolis effect, (ii) is accurate in adverse pressure gradients such as those near wind turbine blades, and (iii) has sufficiently low computational cost to simulate multiple turbines while resolving the blade BL. For simplicity and consistency we develop AMS to be used throughout the domain rather than in a zonal method. We implement our work in the massively parallel flow solver, Nalu-Wind, so that our model can access the compute resources needed for blade-resolved simulations of multiple wind turbines. To accomplish these aims, we modify the baseline AMS by changing the RANS contribution to SST k - omega with a length scale limiter, adding the Coriolis effect, and developing an appropriate wall treatment. We show that AMS of the ABL with the Coriolis effect matches LES reference results better than those obtained with RANS. We describe our plans to add buoyancy effects and wind turbines to our AMS simulations.

atmospheric boundary layer↗

Homotopy Solver

This software implements parallel versions of an interior-point solver, based on the publicly available ipopt solver. Here we have full control over the linear solver and our algorithm is fully parallel thus enabling scalability to large-scale optimization problems. This package also has a parallel implementation of a homotopy solver developed under the scalable methods for contact LDRD project 23-ERD-017. This solver is an mfem-based implementation of algorithm described in ``A filter trust-region Newton continuation method for nonlinear complementarity problems''. Cosmin G. Petra, Nai-Yuan Chiang, Jingyi Wang, Tucker Hartland, and Michael Puso (submitted), LLNL-JRNL-869761.

Hartland, Tucker [Lawrence Livermore National Labo↗

Preparing sparse solvers for exascale computing

Sparse solvers provide essential functionality for a wide variety of scientific applications. Highly parallel sparse solvers are essential for continuing advances in high-fidelity, multi-physics and multi-scale simulations, especially as we target exascale platforms. This paper describes the challenges, strategies and progress of the US Department of Energy Exascale Computing project towards providing sparse solvers for exascale computing platforms. We address the demands of systems with thousands of high-performance node devices where exposing concurrency, hiding latency and creating alternative algorithms become essential. The efforts described here are works in progress, highlighting current success and upcoming challenges.

97 MATHEMATICS AND COMPUTING↗

Progress on Optimizing Wind Farms and Rotor Designs Using Adjoints

Modern wind plants are increasingly tasked with multiple performance objectives. In addition to designing plants that maximize power output and minimize the levelized cost of energy (LCOE), the design and operation of wind plants is increasingly influenced by challenges regarding grid integration of variable generation renewables. This places a growing emphasis on making wind plants more controllable and predictable. WindSE is a Reynolds-averaged Navier-Stokes (RANS) model designed around analytical gradient and adjoint methods, with the ability to capture terrain-induced effects, as shown in Figure 1. The recent addition of an unsteady solver with an actuator line method (ALM) and ongoing work to enable massively parallel optimizations gives it a unique niche to explore coupled plant-level controls and design problems. This code is an open source python package built on the FEniCS framework that utilizes fast, parallel PETSc solvers to model fluid flow throughout wind-farm scale domains. Two recent studies performed using WindSE demonstrate the capability to optimize under a wide variety of flow conditions and objective functions. In the first, we present an optimization focused on modifying the layout of a wind farm with a fixed number of turbines for maximum total power output [1]. This study highlights the ability to quickly perform simulations using the steady Navier-Stokes solver combined with rotors represented as actuator disks while also stressing the importance of capturing terrain-induced effects. Gradient-based optimization using the RANS equations is viable due to the inclusion of efficiently computed adjoint derivatives. We interpret the physical results of the optimal layout and also discuss the computational cost of scaling to larger problems. In the second study, we present the capabilities of the unsteady Navier-Stokes solver, where rotor-blade profiles represented by actuator lines are optimized to enhance wake steering effects and overall power production [2]. We quantify the wind plant performance gains obtained from this type of simultaneous control co-design optimization as compared to optimizing the blade design and yaw independently. Figure 2 shows the differences between a baseline two-turbine system and an optimized system where we fine-tune the blade chord profile. Results and challenges from each study are quickly summarized and used to motivate the current development efforts within WindSE. Current and future work is focused on enabling higher-resolution studies with more degrees of freedom through parallelization of both the simulation and optimization algorithms. We present benchmarking results to show that WindSE performs well in both weak- and strong-scaling tests and further demonstrate that the optimizer obtains the same convergence rates in both shared- and distributed-memory environments. Using larger wind farms, we can study deep-array effects within an optimization context, allowing the use of objective functions that have been previously unstudied. As an example, we present ongoing work on a blockage metric which characterizes the loss of available kinetic energy due to wake effects from multiple upstream turbines.

adjoint optimization↗

Scalable Asynchronous Domain Decomposition Solvers

We discuss how parallel implementations of linear iterative solvers generally alternate between phases of data exchange and phases of local computation. Increasingly large problem sizes and more heterogeneous compute architectures make load balancing and the design of low latency network interconnects that are able to satisfy the communication requirements of linear solvers very challenging tasks. In particular, global communication patterns such as inner products become increasingly limiting at scale. We explore the use of asynchronous communication based on one-sided Message Passing Interface primitives in the context of domain decomposition solvers. In particular, a scalable asynchronous two-level Schwarz method is presented. We discuss practical issues encountered in the development of a scalable solver and show experimental results obtained on a state-of-the-art supercomputer system that illustrate the benefits of asynchronous solvers in load balanced as well as load imbalanced scenarios. Using the novel method, we can observe speedups of up to four times over its classical synchronous equivalent.

97 MATHEMATICS AND COMPUTING↗

PETSc Users Manual (Rev. 3.13)

This manual describes the use of PETSc for the numerical solution of partial differential equations and related problems on high-performance computers. The Portable, Extensible Toolkit for Scientific Computation (PETSc) is a suite of data structures and routines that provide the building blocks for the implementation of large-scale application codes on parallel (and serial) computers. PETSc uses the MPI standard for all message-passing communication. PETSc includes an expanding suite of parallel linear solvers, nonlinear solvers, and time integrators that may be used in application codes written in Fortran, C, C++, and Python. PETSc provides many of the mechanisms needed within parallel application codes, such as parallel matrix and vector assembly routines. The library is organized hierarchically, enabling users to employ the level of abstraction that is most appropriate for a particular problem. By using techniques of object-oriented programming, PETSc provides enormous flexibility for users. PETSc is a sophisticated set of software tools; as such, for some users it initially has a much steeper learning curve than a simple subroutine library. In particular, for individuals without some computer science background, experience programming in C, C++, python, or Fortran and experience using a debugger such as gdb or dbx, it may require a significant amount of time to take full advantage of the features that enable efficient software use. However, the power of the PETSc design and the algorithms it incorporates may make the efficient implementation of many application codes simpler than \rolling them" yourself.

97 MATHEMATICS AND COMPUTING↗

A New Class of AMG Interpolation Methods Based on Matrix-Matrix Multiplications

A new class of distance-two interpolation methods for algebraic multigrid (AMG) that can be formulated in terms of sparse matrix-matrix multiplications is presented and analyzed. Compared with similar distance-two prolongation operators, the proposed algorithms exhibit improved efficiency and portability to various computing platforms, since they allow one to easily exploit existing high-performance sparse matrix kernels. The new interpolation methods have been implemented in hypre, a widely used parallel multigrid solver library. With the proposed interpolations, the overall time of hypre's BoomerAMG setup can be considerably reduced, while sustaining equivalent, sometimes improved, convergence rates. Numerical results for a variety of test problems on parallel machines are presented that support the superiority of the proposed interpolation operators over the existing ones in hypre.

97 MATHEMATICS AND COMPUTING↗

SERGHEI (SERGHEI-SWE) v1.0: a performance-portable high-performance parallel-computing shallow-water solver for hydrology and environmental hydraulics

The Simulation EnviRonment for Geomorphology, Hydrodynamics, and Ecohydrology in Integrated form (SERGHEI) is a multi-dimensional, multi-domain, and multi-physics model framework for environmental and landscape simulation, designed with an outlook towards Earth system modelling. At the core of SERGHEI's innovation is its performance-portable high-performance parallel-computing (HPC) implementation, built from scratch on the Kokkos portability layer, allowing SERGHEI to be deployed, in a performance-portable fashion, in graphics processing unit (GPU)-based heterogeneous systems. In this work, we explore combinations of MPI and Kokkos using OpenMP and CUDA backends. In this contribution, we introduce the SERGHEI model framework and present with detail its first operational module for solving shallow-water equations (SERGHEI-SWE) and its HPC implementation. This module is designed to be applicable to hydrological and environmental problems including flooding and runoff generation, with an outlook towards Earth system modelling. Its applicability is demonstrated by testing several well-known benchmarks and large-scale problems, for which SERGHEI-SWE achieves excellent results for the different types of shallow-water problems. Finally, SERGHEI-SWE scalability and performance portability is demonstrated and evaluated on several TOP500 HPC systems, with very good scaling in the range of over 20 000 CPUs and up to 256 state-of-the art GPUs.

58 GEOSCIENCES↗

A sensitivity analysis of twinning crystal plasticity finite element model using single crystal and poly crystal Zircaloy

The popularity of crystal plasticity finite element method (CPFEM) models is increasing due to their ability to predict the mechanical response of crystalline materials such as metals and metal alloys more accurately than traditional continuum mechanics models. This is since the crystal plasticity models consider the effect of atomic structure, microstructural morphology, and properties of individual grains. These CPFEM models use a large number of material parameters in order to capture the mesoscale physics which comes with the downside of the tedious calibration process. In this paper, a CPFEM code was developed to include the twinning induced grain reorientation and subsequent crystallographic slip for HPC material. The developed code is incorporated in a large-scale, parallelized nonlinear solver WARP3D. Further, a sensitivity analysis with respect to 22 material parameters was then conducted using single crystal and polycrystal representative volume element (RVE) of Zircaloy material. Loading was applied along five different crystallographic orientations for single crystal RVE and along three directions namely, rolling (RD), transverse (TD), and normal (ND) direction for polycrystal RVE. Results obtained from the sensitivity analysis were used for the calibration of material parameters for Zircaloy. Finally, developed code along with calibrated material parameters was used to investigate the effect of the hydride phase formation in Zircaloy which is a typical case observed for nuclear applications. It was found that the volume fraction of the hydride phase has a significant impact on the mechanical properties of Zircaloy.

36 MATERIALS SCIENCE↗

Fast GPU 3D diffeomorphic image registration

3D image registration is one of the most fundamental and computationally expensive operations in medical image analysis. Here, we present a mixed-precision, Gauss–Newton–Krylov solver for diffeomorphic registration of two images. Our work extends the publicly available CLAIRE library to GPU architectures. Despite the importance of image registration, only a few implementations of large deformation diffeomorphic registration packages support GPUs. Our contributions are new algorithms to significantly reduce the run time of the two main computational kernels in CLAIRE: calculation of derivatives and scattered-data interpolation. Additionally, we deploy (i) highly-optimized, mixed-precision GPU-kernels for the evaluation of scattered-data interpolation, (ii) replace Fast-Fourier-Transform (FFT)-based first-order derivatives with optimized 8th-order finite differences, and (iii) compare with state-of-the-art CPU and GPU implementations. As a highlight, we demonstrate that we can register clinical images in less than 6 s on a single NVIDIA Tesla V100. This amounts to over 20 speed-up over the current version of CLAIRE and over 30 speed-up over existing GPU implementations.

97 MATHEMATICS AND COMPUTING↗

A High-Performance Discrete-Element Framework for Simulating Flow and Jamming of Moisture Bearing Biomass Feedstocks

We developed and verified a high-performance open-source discrete element method (DEM) solver with simultaneously-supported feedstock-specific interaction models, including bonded-sphere, liquid bridge, cohesion, and non-linear contact models. Our solver uses parallel data structures on hybrid central and graphics processing unit (CPU/GPU) architectures, with favorable strong scaling performance observed for large problem sizes comprised of (100 M particles), and 4X single-node GPU speedup. The particles for corn stover feedstock were conceptualized and calibrated based on experimental measurements and results. Sensitivity analyses demonstrate that the mass flow rate from a wedge hopper is governed primarily by moisture content, friction coefficient, and cohesion energy density. The model is used to reproduce experimentally observed hopper jamming results, highlighting that the experimental no-flow trends can only be achieved by using non-spherical particles, liquid bridge and cohesion models, highlighting the importance of using concurrent feedstock specialized models for the effective representation of biomass material handling problems.

bioenergy↗

PLEXUS: A Pattern-Oriented Runtime System Architecture for Resilient Extreme-Scale High-Performance Computing Systems

For high-performance computing (HPC) system designers and users, meeting the myriad challenges of next-generation exascale supercomputing systems requires rethinking their approach to application and system software design. Among these challenges, providing resiliency and stability to the scientific applications in the presence of high fault rates requires new approaches to software architecture and design. As HPC systems become increasingly complex, they require intricate solutions for detection and mitigation for various modes of faults and errors that occur in these large-scale systems, as well as solutions for failure recovery. These resiliency solutions often interact with and affect other system properties, including application scalability, power and energy efficiency. Therefore, resilience solutions for HPC systems must be thoughtfully engineered and deployed.In previous work, we developed the concept of resilience design patterns, which consist of templated solutions based on well-established techniques for detection, mitigation and recovery. In this paper, we use these patterns as the foundation to propose new approaches to designing runtime systems for HPC systems. The instantiation of these patterns within a runtime system enables flexible and adaptable end-to-end resiliency solutions for HPC environments. The paper describes the architecture of the runtime system, named Plexus, and the strategies for dynamically composing and adapting pattern instances under runtime control. This runtime-based approach enables actively balancing the cost-benefit trade-off between performance overhead and protection coverage of the resilience solutions. Based on a prototype implementation of PLEXUS, we demonstrate the resiliency and performance gains achieved by the pattern-based runtime system for a parallel linear solver application.

Hukerikar, Saurabh↗

Common Information Model for Electromagnetic Transients (CIM for EMT): CRADA 533 [Abstract only}

The control and protection functions of inverter-based resources (IBR) have raised concerns with bulk system reliability. Most of the current interest lies with solar photovoltaic generation but increasing amounts of storage would pose the same risks. Newer North American Electric Reliability Corporation (NERC) guidelines call for electromagnetic transient (EMT) studies of IBR and recommend that transmission operators collect distributed energy resource (DER) data to support such modeling. IEEE Standard P2800.1 is defining tests for model parameterization, so good model data should become available from inverter vendors. (EMT studies also apply to large power transformer reliability, and transformer vendors can provide EMT models.) Utilities don’t currently have the rest of the bulk system represented for EMT studies at large scale. An International Electrotechnical Commission (IEC) standard Common Information Model (CIM) provides a way of supporting these detailed models from physical asset data, e.g., conductors, towers, transformer data sheets, control block diagrams, while avoiding software vendor lock-in. CIM-for-EMT, with proposed schema extensions and open-source converters, provides a way to exchange EMT data between organizations and tools. This project leverages Office of Electricity (OE) funding of CIM-for-EMT code base through the GridAPPS-DTM project, and of GridPACKTM (parallelized transmission solver), for interoperability testing in CIM-for-EMT. The project also leverages partner PGSTech investments in EMTP® interoperability with CIM.

24 POWER TRANSMISSION AND DISTRIBUTION↗

PETSc/TAO Users Manual V.3.21

This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) and the Toolkit for Advanced Optimization (TAO) for the numerical solution of partial differential equations (PDEs) and related problems on high-performance computers. PETSc/TAO is a suite of data structures and routines that provide the building blocks for implementing large-scale application codes on parallel (and serial) computers. PETSc uses the MPI standard for all distributed memory communication. PETSc/TAO includes a large suite of parallel linear solvers, nonlinear solvers, time integrators, and optimizers that may be used in application codes written in Fortran, C, C++, and Python (via petsc4py; see Getting Started ). The library is organized hierarchically, enabling users to employ the abstraction level most appropriate for a particular problem. By using techniques of object-oriented programming, PETSc provides enormous flexibility for users.

97 MATHEMATICS AND COMPUTING↗

Evaluation of intracoronary hemodynamics identifies perturbations in vorticity

Coronary artery disease (CAD) is highly prevalent and associated with adverse events. Challenges have emerged in the treatment of intermediate coronary artery stenoses. These lesions are often interrogated with fractional flow reserve (FFR) testing to determine if a stenosis is likely to be causative for ischemia in a cardiac territory. This invasive test requires insertion of a pressure wire into a coronary vessel. Recently computational fluid dynamics (CFD) has been used to noninvasively assess fractional flow reserve in vessels reconstructed from medical imaging data. However, many of these simulations are unable to provide additional information about intravascular hemodynamics, including velocity, endothelial shear stress (ESS), and vorticity. We hypothesized that vorticity, which has demonstrated utility in the assessment of ventricular and aortic diseases, would also be an important hemodynamic factor in CAD. Three-dimensional (3D), patient-specific coronary artery geometries that included all vessels >1 mm in diameter were created from angiography data obtained from 10 patients who underwent diagnostic angiography and FFR testing (n = 9). A massively parallel CFD solver (HARVEY) was used to calculate coronary hemodynamic parameters including pressure, velocity, ESS, and vorticity. These simulations were validated by comparing velocity flow fields from simulation to both velocities derived from in vitro particle image velocimetry and to invasively acquired pressure wire-based data from clinical testing. There was strong agreement between findings from CFD simulations and particle image velocimetry experimental testing (p < 0.01). CFD-FFR was also highly correlated with invasively measured FFR (ρ = 0.77, p = 0.01) with an average error of 5.9 ± 0.1%. CFD-FFR also had a strong inverse correlation with the vorticity (ρ = -0.86, p = 0.001). Simulations to determine the effect of the coronary stenosis on intravascular hemodynamics demonstrated significant differences in velocity and vorticity (both p < 0.05). Further evaluation of an angiographically normal appearing non-FFR coronary vessel in patients with CAD also demonstrated differences in vorticity when compared with FFR vessels (p < 0.05). The use of highly accurate 3D CFD-derived intravascular hemodynamics provides additional information beyond pressure measurements that can be used to calculate FFR. Vorticity is one parameter that is modified by a coronary stenosis and appears to be abnormal in angiographically normal vessels in patients with CAD, highlighting a possible use-case in preventative screening for early coronary disease.

59 BASIC BIOLOGICAL SCIENCES↗

Exploiting Power Flow Manifold to Solve AC Optimal Power Flow

AC optimal power flow has proven difficult to solve with interior point methods on GPUs. This is largely due to challenging linear algebra problems that current state of the art massively parallel linear solvers struggle with. However, the advent of Riemannian optimization techniques and the fact that the power flow equations form a smooth manifold present an alternative approach. In this talk, we present the basics of Riemannian optimization techniques in which optimization is done directly on a manifold. Then we present computational results showing that Riemannian techniques are capable of producing solutions of comparable quality as interior point methods.

AC optimal power flow↗

TEAM Project Review, Year 2

This report summarizes our research activities within the TEAM project between December 2020 and December 2021, funded by the ASCR Advanced Research in Quantum Computing program. During the reporting period the LLNL-MSU team has made progress on several fronts. An overarching goal of the team is to provide a comprehensive suite of software tools that can be used for the Characterize-Optimize-Compute loop needed to implement and execute algorithms on quantum devices. We are concurrently developing lightweight solvers that can be used on desktop computers to find optimal control pulses and to characterize small quantum systems (consisting of a few transmons and cavities). However, desktop computers are insufficient for simulating and characterizing larger quantum systems. We have therefore also developed parallel, distributed memory, simulators and optimization solvers, both for open and closed quantum systems. These parallel solvers have, for example, been used to study quantum optimal control for pure-state preparation, utilizing 1000’s of cores on a modern high-performance computing (HPC) platform.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗