Search NASA⌕ Search

Engineering topics

Gardner, David J.

Publications and source records attributed to Gardner, David J..

SUNDIALS time integrators for exascale applications with many independent systems of ordinary differential equations

Many complex systems can be accurately modeled as a set of coupled time-dependent partial differential equations (PDEs). However, solving such equations can be prohibitively expensive, easily taxing the world’s largest supercomputers. One pragmatic strategy for attacking such problems is to split the PDEs into components that can more easily be solved in isolation. This operator splitting approach is used ubiquitously across scientific domains, and in many cases leads to a set of ordinary differential equations (ODEs) that need to be solved as part of a larger “outer-loop” time-stepping approach. The SUNDIALS library provides a plethora of robust time integration algorithms for solving ODEs, and the U.S. Department of Energy Exascale Computing Project (ECP) has supported its extension to applications on exascale-capable computing hardware. In this paper, we highlight some SUNDIALS capabilities and its deployment in combustion and cosmology application codes (Pele and Nyx, respectively) where operator splitting gives rise to numerous, small ODE systems that must be solved concurrently.

97 MATHEMATICS AND COMPUTING↗

Performance of explicit and IMEX MRI multirate methods on complex reactive flow problems within modern parallel adaptive structured grid frameworks

Large-scale multiphysics simulations are computationally challenging due to the coupling of multiple processes with widely disparate time scales. The advent of exascale computing systems exacerbates these challenges since these systems enable ever-increasing size and complexity. In recent years, there has been renewed interest in developing multirate methods as a means to handle the large range of time scales, as these methods may afford greater accuracy and efficiency than more traditional approaches of using implicit-explicit (IMEX) and low-order operator splitting schemes. However, to date there have been few performance studies that compare different classes of multirate integrators on complex application problems. In this work, we study the performance of several newly developed multirate infinitesimal (MRI) methods, implemented in the SUNDIALS solver package, on two reacting flow model problems built on structured mesh frameworks. The first model revisits prior work on a compressible reacting flow problem with complex chemistry that is implemented using BoxLib but where we now include comparisons between a new explicit MRI scheme with the multirate spectral deferred correction (SDC) methods in the original paper. The second problem uses the same complex chemistry as the first problem, combined with a simplified flow model, but runs at a large spatial scale where explicit methods become infeasible due to stability constraints. Two recently developed IMEX MRI multirate methods are tested. These methods rely on advanced features of the AMReX framework on which the model is built, such as multilevel grids and multilevel preconditioners. The results from these two problems show that MRI multirate methods can offer significant performance benefits on complex multiphysics application problems and that these methods may be combined with advanced spatial discretization to compound the advantages of both.

97 MATHEMATICS AND COMPUTING↗

ARKODE: A Flexible IVP Solver Infrastructure for One-step Methods

We describe the ARKODE library of one-step time integration methods for ordinary differential equation (ODE) initial-value problems (IVPs). In addition to providing standard explicit and diagonally implicit Runge–Kutta methods, ARKODE supports one-step methods designed to treat additive splittings of the IVP, including implicit-explicit (ImEx) additive Runge–Kutta methods and multirate infinitesimal (MRI) methods. We present the role of ARKODE within the SUNDIALS suite of time integration and nonlinear solver libraries, the core ARKODE infrastructure for utilities common to large classes of one-step methods, as well as its use of “time stepper” modules enabling easy incorporation of novel algorithms into the library. Numerical results show example problems of increasing complexity, highlighting the algorithmic flexibility afforded through this infrastructure, and include a larger multiphysics application leveraging multiple algorithmic features from ARKODE and SUNDIALS.

97 MATHEMATICS AND COMPUTING↗

Improving Time Step Convergence in an Atmosphere Model With Simplified Physics: The Impacts of Closure Assumption and Process Coupling

Convergence testing is a common practice in the development of dynamical cores of atmospheric models but is not as often exercised for the parameterization of sub-grid physics. An earlier study revealed that the stratiform cloud parameterizations in several predecessors of the Energy Exascale Earth System Model (E3SM) showed strong time-step sensitivity and slower-than-expected convergence when the model's time step was systematically refined. In this work, a simplified atmosphere model is configured that consists of the spectral-element dynamical core of the E3SM atmosphere model coupled with a large-scale condensation parameterization based on commonly used assumptions. This simplified model also resembles E3SM and its predecessors in the numerical implementation of process coupling and shows poor time-step convergence in short ensemble tests. We present a formal error analysis to reveal the expected time-step convergence rate and the conditions for obtaining such convergence. Numerical experiments are conducted to investigate the root causes of convergence problems. We show that revisions in the process coupling and closure assumption help to improve convergence in short simulations using the simplified model; the same revisions applied to a full atmosphere model lead to significant changes in the simulated long-term climate. This work demonstrates that causes of convergence issues in atmospheric simulations can be understood by combining analyses from physical and mathematical perspectives. Addressing convergence issues can help to obtain a discrete model that is more consistent with the intended representation of the physical phenomena.

54 ENVIRONMENTAL SCIENCES↗

Data for publication Wan et al. (2020, JAMES), Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling

This dataset contains simple condensation mdoel output and CAM4 model output that were used to produce figures in the following paper: Wan, H., Woodward, C. S., Zhang, S., Vogl, C. J., Stinis, P., Gardner, D. J., Rasch, P. J., Zeng, X., Larson, V. E., and Singh, B. (2020): Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling, Journal of Advances in Modeling Earth Systems, https://doi.org/10.1029/2019MS001982

54 ENVIRONMENTAL SCIENCES↗

Data from Wan et al. (2020, JAMES, DOI: 10.1029/2019MS001982)

This data set includes the simple condensation model output and CAM4 model output used to generate figures presented in the following paper: Wan, H., Woodward, C. S., Zhang, S., Vogl, C. J., Stinis, P., Gardner, D. J., Rasch, P. J., Zeng, X., Larson, V. E., and Singh, B. (2020): Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling, Journal of Advances in Modeling Earth Systems, https://doi.org/10.1029/2019MS001982

54 ENVIRONMENTAL SCIENCES↗

SUNDIALS Multiphysics+MPIManyVector Performance Testing

In this report we document performance test results on a SUNDIALS-based multiphysics demonstration application. We aim to assess the large-scale parallel performance of new capabilities that have been added to the SUNDIALS suite of time integrators and nonlinear solvers in recent years under funding from both the Exascale Computing Project (ECP) and the Scientific Discovery through Advanced Scientific (SciDAC) program, specifically: 1. SUNDIALS’ new MPIManyVector module, that allows extreme flexibility in how a solution "vector" is staged on computational resources. 2. ARKode’s new multirate integration module, MRIStep, allowing high-order accurate calculations that subcycle "fast" processes within "slow" ones. 3. SUNDIALS’ new flexible linear solver interfaces, that allow streamlined specification of problem-specific linear solvers. 4. SUNDIALS’ new N_Vector additions of "fused" vector operations (to increase arithmetic intensity) and separation of reduction operations into "local" and "global" versions (to reduce latency by combining multiple reductions into a single MPI_Allreduce call). We anticipate that subsequent reports will extend this work to investigate a variety of other new features, including SUNDIALS’ generic SUNNonlinearSolver interface and accelerator-enabled N Vector modules, and upcoming MRIStep extensions to support custom "fast" integrators (that leverage problem structure) and IMEX integration of the "slow" time scale (to add diffusion).

97 MATHEMATICS AND COMPUTING↗

Implicit–explicit (IMEX) Runge–Kutta methods for non-hydrostatic atmospheric models

Abstract. The efficient simulation of non-hydrostatic atmospheric dynamics requires time integration methods capable of overcoming the explicit stability constraints on time step size arising from acoustic waves. In this work, we investigate various implicit–explicit (IMEX) additive Runge–Kutta (ARK) methods for evolving acoustic waves implicitly to enable larger time step sizes in a global non-hydrostatic atmospheric model. The IMEX formulations considered include horizontally explicit – vertically implicit (HEVI) approaches as well as splittings that treat some horizontal dynamics implicitly. In each case, the impact of solving nonlinear systems in each implicit ARK stage in a linearly implicit fashion is also explored. The accuracy and efficiency of the IMEX splittings, ARK methods, and solver options are evaluated on a gravity wave and baroclinic wave test case. HEVI splittings that treat some vertical dynamics explicitly do not show a benefit in solution quality or run time over the most implicit HEVI formulation. While splittings that implicitly evolve some horizontal dynamics increase the maximum stable step size of a method, the gains are insufficient to overcome the additional cost of solving a globally coupled system. Solving implicit stage systems in a linearly implicit manner limits the solver cost but this is offset by a reduction in step size to achieve the desired accuracy for some methods. Overall, the third-order ARS343 and ARK324 methods performed the best, followed by the second-order ARS232 and ARK232 methods.

54 ENVIRONMENTAL SCIENCES↗

Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model

In this study, explicit Runge–Kutta methods and implicit multistep methods utilizing a Newton–Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. This analysis is performed within the shallow-water dynamical core option of a full climate model code base to enable a wealth of simulations for study, with the aim of informing solver development within the more complete hydrostatic dynamical core used for climate research. The limitations and benefits to using explicit versus implicit methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. Given the performance behavior across the configurations analyzed here, the recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, without considering efficiency gains from a preconditioner. The potential for improvement using a preconditioner is greatest for higher spectral order configurations, where more work is shifted to the linear solver. Finally, initial simulations with OpenACC directives to utilize a Graphics Processing Unit (GPU) when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.

58 GEOSCIENCES↗

Filters for Improvement of Multiscale Data from Atomistic Simulations

Multiscale computational models strive to produce accurate and efficient numerical simulations of systems involving interactions across multiple spatial and temporal scales that typically differ by several orders of magnitude. Some such models utilize a hybrid continuum-atomistic approach combining continuum approximations with first-principles-based atomistic models to capture multiscale behavior. By following the heterogeneous multiscale method framework for developing multiscale computational models, unknown continuum scale data can be computed from an atomistic model. Concurrently coupling the two models requires performing numerous atomistic simulations which can dominate the computational cost of the method. Furthermore, when the resulting continuum data is noisy due to sampling error, stochasticity in the model, or randomness in the initial conditions, filtering can result in significant accuracy gains in the computed multiscale data without increasing the size or duration of the atomistic simulations. In this work, we demonstrate the effectiveness of spectral filtering for increasing the accuracy of noisy multiscale data obtained from atomistic simulations. Moreover, we present a robust and automatic method for closely approximating the optimum level of filtering in the case of additive white noise. By improving the accuracy of this filtered simulation data, it leads to a dramatic computational savings by allowing for shorter and smaller atomistic simulations to achieve the same desired multiscale simulation precision.

97 MATHEMATICS AND COMPUTING↗

On the use of finite difference matrix-vector products in Newton-Krylov solvers for implicit climate dynamics with spectral elements

Efficient solutions of global climate models require effectively handling disparate length and time scales. Implicit solution approaches allow time integration of the physical system with a step size governed by accuracy of the processes of interest rather than by stability of the fastest time scales present. Implicit approaches, however, require the solution of nonlinear systems within each time step. Usually, a Newton's method is applied to solve these systems. Each iteration of the Newton's method, in turn, requires the solution of a linear model of the nonlinear system. This model employs the Jacobian of the problem-defining nonlinear residual, but this Jacobian can be costly to form. If a Krylov linear solver is used for the solution of the linear system, the action of the Jacobian matrix on a given vector is required. In the case of spectral element methods, the Jacobian is not calculated but only implemented through matrix-vector products. The matrix-vector multiply can also be approximated by a finite difference approximation which may introduce inaccuracy in the overall nonlinear solver. In this paper, we review the advantages and disadvantages of finite difference approximations of these matrix-vector products for climate dynamics within the spectral element shallow water dynamical core of the Community Atmosphere Model.

54 ENVIRONMENTAL SCIENCES↗