Search NASA⌕ Search

SEARCH · Search NASA

Results for “gyrokinetic PIC code”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Hybrid-view programming of nuclear fusion simulation code in the PGAS parallel programming language XcalableMP

Recently, the Partitioned Global Address Space (PGAS) parallel programming model has emerged as a usable distributed memory programming model. XcalableMP (XMP) is a PGAS parallel programming language that extends base languages such as C and Fortran with directives in OpenMP-like style. XMP supports a global-view model that allows programmers to define global data and to map them to a set of processors, which execute the distributed global data as a single thread. In XMP, the concept of a coarray is also employed for local-view programming. In this study, we port Gyrokinetic Toroidal Code - Princeton (GTC-P), which is a three-dimensional gyrokinetic PIC code developed at Princeton University to study the microturbulence phenomenon in magnetically confined fusion plasmas, to XMP as an example of hybrid memory model coding with the global-view and local-view programming models. In local-view programming, the coarray notation is simple and intuitive compared with Message Passing Interface (MPI) programming while the performance is comparable to that of the MPI version. Thus, because the global-view programming model is suitable for expressing the data parallelism for a field of grid space data, we implement a hybrid-view version using a global-view programming model to compute the field and a local-view programming model to compute the movement of particles. Finally, the performance is degraded by 20% compared with the original MPI version, but the hybrid-view version facilitates more natural data expression for static grid space data (in the global-view model) and dynamic particle data (in the local-view model), and it also increases the readability of the code for higher productivity.

97 MATHEMATICS AND COMPUTING↗

Orb5: A global electromagnetic gyrokinetic code using the PIC approach in toroidal geometry

This paper presents the current state of the global gyrokinetic code Orb5 as an update of the previous reference [Jolliet et al., Comp. Phys. Commun. 177 409 (2007)]. The Orb5 code solves the electromagnetic VlasovMaxwell system of equations using a PIC scheme and also includes collisions and strong flows. The code assumes multiple gyrokinetic ion species at all wavelengths for the polarization density and drift-kinetic electrons. Variants of the physical model can be selected for electrons such as assuming an adiabatic response or a “hybrid” model in which passing electrons are assumed adiabatic and trapped electrons are drift-kinetic. A Fourier filter as well as various control variates and noise reduction techniques enable simulations with good signal-to-noise ratios at a limited numerical cost. They are completed with different momentum and zonal flow-conserving heat sources allowing for temperature-gradient and flux-driven simulations. The code, which runs on both CPUs and GPUs, is well benchmarked against other similar codes and analytical predictions, and shows good scalability up to thousands of nodes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Optimizing fusion PIC code performance at scale on Cori Phase 2

In this paper we present the results of optimizing the performance of the gyrokinetic full-f fusion PIC code XGC1 on the Cori Phase Two Knights Landing system. The code has undergone substantial development to enable the use of vector instructions in its most expensive kernels within the NERSC Exascale Science Applications Program. We study the single-node performance of the code on an absolute scale using the roofline methodology to guide optimization efforts. We have obtained 2x speedups in single node performance due to enabling vectorization and performing memory layout optimizations. On multiple nodes, the code is shown to scale well up to 4000 nodes, near half the size of the machine. We discuss some communication bottlenecks that were identified and resolved during the work.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Verification of long wavelength electromagnetic modes with a gyrokinetic-fluid hybrid model in the XGC code

As an alternative option to kinetic electrons, the gyrokinetic total-f particle-in-cell (PIC) code XGC1 has been extended to the MHD/fluid type electromagnetic regime by combining gyrokinetic PIC ions with massless drift-fluid electrons. Here, two representative long wavelength modes, shear Alfven waves and resistive tearing modes, are verified in cylindrical and toroidal magnetic field geometries.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Performance of particle in cell methods on highly concurrent computational architectures

Particle in cell (PIC) methods are effective in computing Vlasov-Poisson system of equations used in simulations of magnetic fusion plasmas. PIC methods use grid based computations, for solving Poisson’s equation or more generally Maxwell’s equations, as well as Monte-Carlo type methods to sample the Vlasov equation. The presence of two types of discretizations, deterministic field solves and Monte-Carlo methods for the Vlasov equation, pose challenges in understanding and optimizing performance on today large scale computers which require high levels of concurrency. These challenges arises from the need to optimize two very different types of processes and the interactions between them. Modern cache based high-end computers have very deep memory hierarchies and high degrees of concurrency which must be utilized effectively to achieve good performance. The effective use of these machines requires maximizing concurrency by eliminating serial or redundant work and minimizing global communication. A related issue is minimizing the memory traffic between levels of the memory hierarchy because performance is often limited by the bandwidths and latencies of the memory system. This paper discusses some of the performance issues, particularly in regard to parallelism, of PIC methods. The gyrokinetic toroidal code (GTC) is used for these studies and a new radial grid decomposition is presented and evaluated. Scaling of the code is demonstrated on ITER sized plasmas with up to 16K Cray XT3/4 cores.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Performance of particle in cell methods on highly concurrent computational architectures

Particle in cell (PIC) methods are effective in computing Vlasov-Poisson system of equations used in simulations of magnetic fusion plasmas. PIC methods use grid based computations, for solving Poisson’s equation or more generally Maxwell’s equations, as well as Monte-Carlo type methods to sample the Vlasov equation. The presence of two types of discretizations, deterministic field solves and Monte-Carlo methods for the Vlasov equation, pose challenges in understanding and optimizing performance on today large scale computers which require high levels of concurrency. These challenges arises from the need to optimize two very different types of processes and the interactions between them. Modern cache based high-end computers have very deep memory hierarchies and high degrees of concurrency which must be utilized effectively to achieve good performance. The effective use of these machines requires maximizing concurrency by eliminating serial or redundant work and minimizing global communication. A related issue is minimizing the memory traffic between levels of the memory hierarchy because performance is often limited by the bandwidths and latencies of the memory system. This paper discusses some of the performance issues, particularly in regard to parallelism, of PIC methods. The gyrokinetic toroidal code (GTC) is used for these studies and a new radial grid decomposition is presented and evaluated. Scaling of the code is demonstrated on ITER sized plasmas with up to 16K Cray XT3/4 cores.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Global gyrokinetic particle-in-cell simulations of internal kink instabilities

Internal kink instabilities have been studied in straight tokamak geometry employing an electromagnetic gyrokinetic particle-in-cell (PIC) code. The ideal-MHD internal kink mode and the collisionless m=1 tearing mode have been successfully simulated with the PIC code. Diamagnetic effects on the internal kink modes have also been investigated.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Particle simulation of radio frequency waves with fully-kinetic ions and gyrokinetic electrons

A toroidal particle-in-cell (PIC) code, suitable for investigating nonlinear phenomena in radio frequency (RF) heating and current drive, is developed and verified thereafter through a series of fidelity tests for field solvers and single particle motions in toroidal geometry, where simulation results show good coincidence with analytical prediction. The RF capability is then demonstrated through the integrated benchmarks with linear lower hybrid wave and ion Bernstein wave theory in cylindrical geometry, where the analytic result is easily available. The frequency and mode structure in the simulations agree well with the theoretical prediction.

Lin, Jingbo↗

Memory-efficient optimization of Gyrokinetic particle-to-grid interpolation for multicore processors

We present multicore parallelization strategies for the particle-to-grid interpolation step in the Gyrokinetic Toroidal Code (GTC), a 3D particle-in-cell (PIC) application to study turbulent transport in magnetic-confinement fusion devices. Particle-grid interpolation is a known performance bottleneck in several PIC applications. In GTC, this step involves particles depositing charges to a 3D toroidal mesh, and multiple particles may contribute to the charge at a grid point. We design new parallel algorithms for the GTC charge deposition kernel, and analyze their performance on three leading multicore platforms. We implement thirteen different variants for this kernel and identify the best-performing ones given typical PIC parameters such as the grid size, number of particles per cell, and the GTC-specific particle Larmor radius variation. We find that our best strategies can be 2x faster than the reference optimized MPI implementation, and our analysis provides insight into desirable architectural features for high-performance PIC simulation codes.

97 MATHEMATICS AND COMPUTING↗

Center for Gyrokinetic/MHD Hybrid Simulation of Energetic Particle Physics in Toroidal Plasmas (Final Report)

At Colorado University-Boulder the primary task is to extend our gyrokinetic Particle-in-Cell simulation of tokamak micro-turbulence and transport to the area of energetic particle physics. We have implemented a gyrokinetic ion/massless fluid electron hybrid model in the global δƒ-PIC code GEM, and benchmarked the code with analytic results on the thermal ion radiative damping rate of Toroidal Alfvén Eigenmodes (TAE) and with mode frequency and spatial structure from eigenmode analysis. We also performed nonlinear simulations of both a single-n mode (n is the toroidal mode number) and multiple-n modes, and in the case of single-n, benchmarked the code on the saturation amplitude vs. particle collision rate with analytical theory. Most simulations use the f method for both ions species, but we have explored the full-ƒ method for energetic particles in cases where the burst amplitude of the excited instabilities is large as to cause significant re-distribution or loss of the energetic particles. We used the hybrid model to study the stability of high-n TAEs in ITER. Our simulations show that the most unstable modes in ITER lie in the rage of 10 < n < 20. Thermal ion pressure effect and alpha particles non-perturbative effect are important in determining the mode radial location and stability threshold. The thermal ion Landau damping rate and radiative damping rate from the simulations are compared with analytical estimates. The thermal ion Landau damping is the dominant damping mechanism. Plasma elongation has a strong stabilizing effect on the alpha driven TAEs. The central alpha particle pressure threshold for the most unstable n=15 mode is about β α (0) = 0.7% for the fully shaped ITER equilibrium. We also carried nonlinear simulations of the most unstable n = 15 mode and found that the saturation amplitude for the nominal ITER discharge is too low to cause large redistribution or loss of alpha particles. To include kinetic electron effects in the hybrid model we have studied a kinetic electron closure scheme for the fluid electron model. The most important element of the closure scheme is a complete Ohm’s law for the parallel electric field E || , derived by combining the quasi-neutrality condition, the Ampere’s equation and the v || moment of the gyrokinetic equations. A discretization method for the closure scheme is studied in detail for a three-dimensional shear-less slab plasma. It is found that for long-wavelength shear Alfvén waves the kinetic closure scheme is both more accurate and robust than the previous GEM algorithm using the split-scheme, whereas for the ion-gradient-driven instability the previous algorithm is more efficient. This kinetic electron closure scheme will be implemented in GEM in the future. We have studied the beam driven Reverse Shear Alfven Eigenmodes (RSAE) observed in DIII-D discharge 142111. For this purpose a new scheme for obtaining the electric potential is implemented, i.e., by solving the gyrokinetic moment (GKM) equation, which is essentially the equation for $∂\phi/∂t$ used in GEM’s split-weight scheme, and then integrating in time. Due to charge-neutrality the ExB motions of the equilibrium densities of all species cancel each other and do not cause charge separation if there is no finite Larmor radius effect. The advantage of solving the GKM equation is that this lowest-order cancellation can be made explicit. The GKM approach is found to be more accurate and robust. GEM simulations have reproduced many features of RSAE seen in the experiment, such as frequency chirping and the chirping range. It has been reported by other simulation codes that the shearing direction of the mode structure in the poloidal plane disagrees with observation. We found that the mode structure, including the shearing in the poloidal plane, is in general sensitive to the beam distribution. Using the same beam density profile as in other codes but with a slowing-down distribution in velocity, GEM simulations reproduce the mode shearing direction seen in the experiment. We have carried out extensive nonlinear simulations of RSAE.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers

The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and particle decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method to extreme computational scales. In this paper, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputing systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects and network topologies. Furthermore, these performance comparisons using a realistic discovery-science-capable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Gyrokinetic toroidal simulations on leading multi- and manycore HPC systems

The gyrokinetic Particle-in-Cell (PIC) method is a critical computational tool enabling petascale fusion simulation re-search. In this work, we present novel multi- and manycore-centric optimizations to enhance performance of GTC, a PIC-based production code for studying plasma microtur-bulence in tokamak devices. Our optimizations encompass all six GTC sub-routines and include multi-level particle and grid decompositions designed to improve multi-node parallel scaling, particle binning for improved load balance, GPU acceleration of key subroutines, and memory-centric optimizations to improve single-node scaling and reduce memory utilization. The new hybrid MPI-OpenMP and MPI-OpenMP-CUDA GTC versions achieve up to a 2× speedup over the production Fortran code on four parallel systems - clusters based on the AMD Magny-Cours, Intel Nehalem-EP, IBM BlueGene/P, and NVIDIA Fermi architectures. Finally, strong scaling experiments provide insight into parallel scalability, memory utilization, and programmability trade-offs for large-scale gyrokinetic PIC simulations, while attaining a 1.6× speedup on 49,152 XE6 cores.

97 MATHEMATICS AND COMPUTING↗

Collaborative Research: Particle Simulation of Collisionless Magnetic Reconnection Under Finite Guide Field (Final Technical Report)

The project investigated 3-D linear instabilities of Harris current sheet under a realistic ion-to-electron mass ratio and a finite guide field. The excitation and 3-D properties of lower hybrid drift instability (LHDI), along with two new instabilities were demonstrated. The report presents the nonlinear properties of current sheet instabilities by employing Gyrokinetic electron and Fully kinetic ion (GeFi) particle simulation code. The GeFi simulation results are compared with fully kinetic PIC simulations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Applying the Gyrokinetic Formulation of Magnetized Plasma Dynamics to Z-pinch plasmas

This project adapts and applies the gyrokinetic Eulerian code COGENT to an emerging area of highly-magnetized Z-pinch plasma systems. Simulation efforts for these plasmas currently use either magneto-hydrodynamic (MHD) fluid codes, which suffer from fidelity being restricted to narrow plasma conditions, or fully kinetic particle-in-cell (PIC) methods, which suffer from the run-time needed to resolve the very small spatial and temporal scales required for the plasma conditions. The progress with achieving detailed theoretical understanding of Z-pinch plasma systems can be greatly expedited using some combination of gyrokinetic and drift-ordered MHD models, developed in the tokamak community and implemented in the COGENT code. These formulations are applicable to a much larger parameter space than standard MHD models, while providing a substantial speed-up in run-time over fully kinetic codes. In particular, the gyrokinetic formalism is successfully applied to a flow-stabilized Z pinch (FSZP) system – a promising fusion concept where a sheared axial flow is used to provide stability against large-scale ideal-MHD modes. Results of gyrokinetic simulations with COGENT agree well with those from fully kinetic simulations, while requiring ~500x less run-time.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

CICART Center For Integrated Computation And Analysis Of Reconnection And Turbulence

CICART is a partnership between the University of New Hampshire (UNH) and Dartmouth College. CICART addresses two important science needs of the DoE: the basic understanding of magnetic reconnection and turbulence that strongly impacts the performance of fusion plasmas, and the development of new mathematical and computational tools that enable the modeling and control of these phenomena. The principal participants of CICART constitute an interdisciplinary group, drawn from the communities of applied mathematics, astrophysics, computational physics, fluid dynamics, and fusion physics. It is a main premise of CICART that fundamental aspects of magnetic reconnection and turbulence in fusion devices, smaller-scale laboratory experiments, and space and astrophysical plasmas can be viewed from a common perspective, and that progress in understanding in any of these interconnected fields is likely to lead to progress in others. The establishment of CICART has strongly impacted the education and research mission of a new Program in Integrated Applied Mathematics in the College of Engineering and Applied Sciences at UNH by enabling the recruitment of a tenure-track faculty member, supported equally by UNH and CICART, and the establishment of an IBM-UNH Computing Alliance. The proposed areas of research in magnetic reconnection and turbulence in astrophysical, space, and laboratory plasmas include the following topics: (A) Reconnection and secondary instabilities in large high-Lundquist-number plasmas, (B) Particle acceleration in the presence of multiple magnetic islands, (C) Gyrokinetic reconnection: comparison with fluid and particle-in-cell models, (D) Imbalanced turbulence, (E) Ion heating, and (F) Turbulence in laboratory (including fusion-relevant) experiments. These theoretical studies make active use of three high-performance computer simulation codes: (1) The Magnetic Reconnection Code, based on extended two-fluid (or Hall MHD) equations, in an Adaptive Mesh Refinement (AMR) framework, (2) the Particle Simulation Code, a fully electromagnetic 3D Particle-In-Cell (PIC) code that includes a collision operator, and (3) GS2, an Eulerian, electromagnetic, kinetic code that is widely used in the fusion program, and simulates the nonlinear gyrokinetic equations, together with a self-consistent set of Maxwell’s equations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Re-splitting δ f method for electro-magnetic gyrokinetic particle-in-cell (PIC) simulation of tokamak plasmas

A new method based on a re-splitting δ f technique is proposed to mitigate the cancellation problem in electro-magnetic gyrokinetic PIC simulation. This new scheme, together with the pull-back mitigation (PBM) scheme (Mishchenko et al., 2014), have recently been implemented in the GEM code (Chen et al., 2003). These two algorithms are systematically compared with the original GEM algorithm for the simulation of the ion temperature gradient (ITG) mode, the kinetic ballooning mode (KBM), the toroidal Alfvén eigenmode (TAE) and the energetic particle mode (EPM). The three algorithms agree well for all of these modes. Moreover, both the re-splitting method and PBM can mitigate the cancellation problem and improve the computational efficiency.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Benchmarking gyrokinetic simulations in a toroidal flux-tube

A flux-tube model is implemented in the global turbulence code GEM [Y. Chen and S. E. Parker, J. Comput. Phys. 220, 839 (2007)] in order to facilitate benchmarking with Eulerian codes. The global GEM assumes the magnetic equilibrium to be completely given. The initial flux-tube implementation simply selects a radial location as the center of the flux-tube and a radial size of the flux-tube, sets all equilibrium quantities (B, ∇B, etc.) to be equal to the values at the center of the flux-tube, and retains only a linear radial profile of the safety factor needed for boundary conditions. This implementation shows disagreement with Eulerian codes in linear simulations. An alternative flux-tube model based on a complete local equilibrium solution of the Grad-Shafranov equation [J. Candy, Plasma Phys. Controlled Fusion 51, 105009 (2009)] is then implemented. This results in better agreement between Eulerian codes and the particle-in-cell (PIC) method. The PIC algorithm based on the v{sub ||}-formalism [J. Reynders, Ph.D. dissertation, Princeton University, 1992] and the gyrokinetic ion/fluid electron hybrid model with kinetic electron closure [Y. Chan and S. E. Parker, Phys. Plasmas 18, 055703 (2011)] are also implemented in the flux-tube geometry and compared with the direct method for both the ion temperature gradient driven modes and the kinetic ballooning modes.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Gyrokinetic particle-in-cell optimization on emerging multi- and manycore platforms

The next decade of high-performance computing (HPC) systems will see a rapid evolution and divergence of multi- and manycore architectures as power and cooling constraints limit increases in microprocessor clock speeds. Understanding efficient optimization methodologies on diverse multicore designs in the context of demanding numerical methods is one of the greatest challenges faced today by the HPC community. In this paper, we examine the efficient multicore optimization of GTC, a petascale gyrokinetic toroidal fusion code for studying plasma microturbulence in tokamak devices. For GTC’s key computational components (charge deposition and particle push), we explore efficient parallelization strategies across a broad range of emerging multicore designs, including the recently-released Intel Nehalem-EX, the AMD Opteron Istanbul, and the highly multithreaded Sun UltraSparc T2+. We also present the first study on tuning gyrokinetic particle-in-cell (PIC) algorithms for graphics processors, using the NVIDIA C2050 (Fermi). Our work discusses several novel optimization approaches for gyrokinetic PIC, including mixed-precision computation, particle binning and decomposition strategies, grid replication, SIMDized atomic floating-point operations, and effective GPU texture memory utilization. Overall, we achieve significant performance improvements of 1.3–4.7× on these complex PIC kernels, despite the inherent challenges of data dependency and locality. Finally, our work also points to several architectural and programming features that could significantly enhance PIC performance and productivity on next-generation architectures.

97 MATHEMATICS AND COMPUTING↗