Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel implementation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

On the adequacy of lightweight thread approaches for high-level parallel programming models

High-level parallel programming models (PMs) are becoming crucial in order to extract the computational power of current on-node multi-threaded parallelism. The most popular PMs, such as OpenMP or OmpSs, are directive-based: the complexity of the hardware is hidden by the underlying runtime system, improving coding productivity. The implementations of OpenMP usually rely on POSIX threads (pthreads), offering excellent performance for coarse-grained parallelism and a perfect match with the current hardware. OmpSs is a task oriented PM based on an ad hoc runtime solution called Nanos++; it is the precursor of the tasking parallelism in the OpenMP tasking specification. A recent trend in runtimes and applications points to leveraging massive on-node parallelism in conjunction with fine-grained and dynamic scheduling paradigms. In this paper we analyze the behavior of the OpenMP and OmpSs PMs on top of the recently emerged Generic Lightweight Threads (GLT) API. GLT exposses a common API for lightweight thread (LWT) libraries that offers the possibility of running the same application over different native LWT solutions. We describe the design details of those high-level PMs implemented on top of GLT and analyze different scenarios in order to assess where the use of LWTs may benefit application performance. Our work reveals those scenarios where LWTs overperform pthread-based solutions and compares the performance between an ad hoc solution and a generic implementation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

On the Degradation of the Effectiveness of Nonlinear Diffusion Acceleration with Parallel Block Jacobi Splitting

MOOSE's radiation transport module Rattlesnake comprises a first order S N solver that can be accelerated with a flexible Nonlinear Diffusion Acceleration. At each NDA iteration, the streaming and collision operator of the S{sub N} equations are (approximately) inverted using a transport sweep. When executed on multiple processors parallel block Jacobi (PBJ) splitting is utilized lagging the processor sub-domain's incoming angular fluxes making the implemented method asynchronous, i.e. the convergence properties of NDA become dependent on the number of concurrent tasks and details of the partitioning. Within this work, we study the NDA convergence properties for the two-dimensional C5G7 benchmark problem PBJ with p = 1, 2, 4, 8, 12, 16, 24, 32, 48, 60, 72, 96 and 144 concurrent tasks. Rosa studied the convergence properties of PBJ splitting methods with and without transport synthetic acceleration (TSA) in slab geometry and two-dimensional Cartesian geometry, respectively. Rosa's iteration scheme lags both the incoming angular fluxes and the scattering source making it very similar to the method described here. Rosa found by means of Fourier analysis that convergence of unaccelerated PBJ can degrade for optically thin subdomains, i.e. for a large number of processors given a fixed domain size. Depending on the choice of parameters application of TSA can lead to an effective algorithm. In contrast to Rosa's work, this work focuses on PBJ splitting for NDA accelerated solution of the multigroup eigenvalue problem. Later, Zerr and Anistratov used their integral transport matrix method in conjunction with PBJ. This iteration scheme is different as it does not lag the scattering source. Nevertheless, Fourier analysis with Diamond Difference spatial discretization showed that the method degrades with increasing the overall size of the spatial domain while keeping the sub-domain size constant. In the case of an infinite medium, the observed spectral radius becomes unity. These references establish the iterative performance of PBJ for one-group fixed source problems discretized on rectangular cells, but for the Rattlesnake first order NDA method, an emphasis lies on multigroup eigenvalue problems. In this setting, the additional complication arises that the low order equations are not exactly inverted, but only to some iterative criterion. Further, as the low order problem is solved using Jacobian-Free Newton Krylov methods, a suitable preconditioner for the GMRES iterations is required. As the effectiveness of the preconditioners may decrease when more concurrent processes are used, we expect the low order solve to impact the overall effectiveness of the NDA method in parallel execution as well. Therefore, this work focuses on the question of how detrimental the asynchronicity of the high and low order iterative solutions is for the effectiveness of NDA solutions of realistic, unstructured grid, multigroup eigenvalue problems. The iterative convergence properties of Rattlesnake's NDA method for multigroup eigenvalue problems are studied for varying number of concurrent processes. Rattlesnake's NDA method is asynchronous because the high order transport sweep uses PBJ splitting and because the low order solve's preconditioners, algebraic multigrid and additive Schwarz, are asynchronous. We found that up to 144 processes the number of Picard iterations increases by a factor of 2-3 affecting but not destroying NDA's parallel performance for the C5G7 benchmark problem, a representative LWR (light water reactor) configuration. The solution of the NDA low order eigenvalue problem posed greater problems than expected for the solvers currently available in Rattlesnake mainly because of problems to effectively precondition the modified low-order problem. As a matter of fact, the low order problem becomes an advection-diffusion problem. Non-robustness was observed for the AMG preconditioned coarse mesh acceleration for more than 60 concurrent processes, which was fixed by increasing the number of permissible V-cycles per AMG call. For fine mesh acceleration AMG proved to be ineffective and we resorted to ASM. For both AMG and ASM, increasing number of linear iterations per nonlinear are observed when increasing the number of processes, but for ASM this problem appears to be worse. The results from this work also indicate that initial development efforts should be diverted from implementation of a high-order parallel sweeping algorithm to the effective and robust iterative solution of the low order NDA multigroup eigenvalue problem. (authors)

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

TU-AB-202-05: GPU-Based 4D Deformable Image Registration Using Adaptive Tetrahedral Mesh Modeling

Purpose: Deformable image registration (DIR) has been employed today as an automated and effective segmentation method to transfer tumor or organ contours from the planning image to daily images, instead of manual segmentation. However, the computational time and accuracy of current DIR approaches are still insufficient for online adaptive radiation therapy (ART), which requires real-time and high-quality image segmentation, especially in a large datasets of 4D-CT images. The objective of this work is to propose a new DIR algorithm, with fast computational speed and high accuracy, by using adaptive feature-based tetrahedral meshing and GPU-based parallelization. Methods: The first step is to generate the adaptive tetrahedral mesh based on the image features of a reference phase of 4D-CT, so that the deformation can be well captured and accurately diffused from the mesh vertices to voxels of the image volume. Subsequently, the deformation vector fields (DVF) and other phases of 4D-CT can be obtained by matching each phase of the target 4D-CT images with the corresponding deformed reference phase. The proposed 4D DIR method is implemented on GPU, resulting in significantly increasing the computational efficiency due to its parallel computing ability. Results: A 4D NCAT digital phantom was used to test the efficiency and accuracy of our method. Both the image and DVF results show that the fine structures and shapes of lung are well preserved, and the tumor position is well captured, i.e., 3D distance error is 1.14 mm. Compared to the previous voxel-based CPU implementation of DIR, such as demons, the proposed method is about 160x faster for registering a 10-phase 4D-CT with a phase dimension of 256×256×150. Conclusion: The proposed 4D DIR method uses feature-based mesh and GPU-based parallelism, which demonstrates the capability to compute both high-quality image and motion results, with significant improvement on the computational speed.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

Parallel Execution of Functional Mock-up Units in Buildings Modeling

A Functional Mock-up Interface (FMI) defines a standardized interface to be used in computer simulations to develop complex cyber-physical systems. FMI implementation by a software modeling tool enables the creation of a simulation model that can be interconnected, or the creation of a software library called a Functional Mock-up Unit (FMU). This report describes an FMU wrapper implementation that imports FMUs into a C++ environment and uses an Euler solver that executes FMUs in parallel using Open Multi-Processing (OpenMP). The purpose of this report is to elucidate the runtime performance of the solver when a multi-component system is imported as a single FMU (for the whole system) or as multiple FMUs (for different groups of components as sub-systems). This performance comparison is conducted using two test cases: (1) a simple, multi-tank problem; and (2) a more realistic use case based on the Modelica Buildings Library. In both test cases, the performance gains are promising when each FMU consists of a large number of states and state events that are wrapped in a single FMU. Load balancing is demonstrated to be a critical factor in speeding up parallel execution of multiple FMUs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Data Analytics for Electrical Distribution Systems with Micro PMUs (GMLC 1.4.9 Technical Report)

This report describes how electrical utilities with residential, industrial, commercial and government customers make decisions using Data Analytics (DA) applications and discusses how OpenFMB was leveraged to share that data among disparate types of devices for DA. OpenFMB is a data standard developed by utilities for distributed communication and control of power system assets. Here we explored how OpenFMB can be used for sharing analytics and what those analytics could be for a set of representative grid devices. A review of DA applications performed by electrical utilities and customers is presented. An overview of how the testbed we used was built and configured is summarized. Several experiments which were carried out to explore the capabilities of OpenFMB alongside other data sharing mechanisms for gathering sensor measurements and sharing analytics is also discussed. And finally a comparison of OpenFMB and LBNL’s SPARK framework is summarized. Today’s electrical utilities focus on monitoring the real-time customer consumptions, offering new real-time tariff models based on customer usage habits (rewards and punishments), providing energy peak during peak load times (plan energy generation and distribution), improving the load management for transformers and power lines, and controlling the energy and grid resources efficiently. On the other hand, the DA applications for residential customers are based on modifying the customer electricity usage habit to save money, making accurate and timely decisions for electrical billing. However, the DA applications for industrial, commercial and government customers focus on creating efficient programs to improve the environmental condition (reducing CO₂ emission), developing better load forecasting for industrial and/or commercial activities. In this project, a testbed was built to explore how the OpenFMB framework can be used to distribute data among various types of representative grid edge devices. Micro PMUs (Phasor Measurement Unit) were used to collect and transmit data to an OpenPDC server alongside relays with a framework configured with a system using OpenFMB to collect and transmit data. The testbed was composed of a physical asset based on a 60 Hz AC radial power system constructed using SI-GRID components. The power system consisted of a 3-phase 14 volts line-to-ground (24 volts line-to-line) power source, three 3-phase line emulators, a programmable resistive load bank, two smart relays, two micro PMUs, and a (SCADA) and Phasor Data Concentrator (PDC) computer. This testbed configuration allowed us to perform breaker operation tests collecting data from the micro PMUs and from the relays’ sensors. During the micro PMU breaker operation tests we collected the measured and estimated fundamental voltages, currents, frequency as well as the total true, reactive and apparent power. The displacement power factor from two micro PMUs located in parallel power lines was also used. The software steps for the implementation of the Synchrophasor system are explained in detail in this report. The setting (1-2-3) and data collection (4-5) phases were performed with a free download open source software. In the setting phase, the micro PMUs were set (1), the data stream for each micro PMU was validated, and the connection and configuration files were created (2). The PDC computer was set using the connection and configuration files of the micro PMUs (3). In the data collection phase, the binary data from the micro PMUs was collected (4), and the binary data files of the micro PMUs was converted to analyze the test results (5). OpenFMB was configured to use protocol translators to read Modbus and transform the reading into the OpenFMB data model to be read for DA applications.Our experiments showed that by leveraging the OpenFMB data model and the micro PMUs together we could obtain better situational awareness of network faults, which in some cases could lead to failures in the system and would be less perceivable otherwise. For example, in one experiment the fundamental RMS current angle, and the fundamental displacement power factor (estimated by the ratio between the measured real and apparent total power) had a disturbance at the micro PMU connected to the beginning of the parallel power line because the breaker was opened without the operators or SCADA system’s knowledge, and no current was available. Based on the breaker state scenarios and micro PMU locations, the DA applications using the current angle and displacement power factor from the micro PMUs should be studied in detail to avoid a non-desired measurement that could affect the decision from the DA. Also, the sample rates for PMU, Relay and micro PMU devices were compared. The DA application improved the control and operation between the electrical utilities and their customers (residential, industrial, transportation, commercial and government). The higher the sample rate, the better the quality of the measurement for DA estimations.

42 ENGINEERING↗
DOE OSTI2019

Machine‐learning‐based predictive control of nonlinear processes. Part II: Computational implementation

Abstract Machine learning is receiving more attention in classical engineering fields, and in particular, recurrent neural networks (RNNs) coupled with ensemble regression tools have demonstrated the capability of modeling nonlinear dynamic processes. In Part I of this two‐article series, the Lyapunov‐based model predictive control (LMPC) method using a single RNN model and an ensemble of RNN models, respectively, was rigorously developed for a general class of nonlinear systems. In the present article, computational implementation issues of this new control method ranging from training of the RNN models, ensemble regression of the RNN models, and parallel computation for accelerating the real‐time model calculations are addressed. Furthermore, a chemical reactor example is used to demonstrate the implementation and effectiveness of these machine‐learning tools in LMPC as well as compare them with standard state‐space model identification tools.

Wu, Zhe↗
DOE OSTI2022

Ion temperature anisotropy model with cross‐field drifts in the scrape‐off layer

Abstract An ion temperature anisotropy model including cross‐field drifts has been implemented in the 2D transport code UEDGE. Two ion temperature equations are solved in the model to give separate ion temperatures perpendicular and parallel to the magnetic field, which results in an additional mirror force for ion parallel transport and also leads to an additional curvature drift in the perpendicular direction. In our previous work, an ion temperature anisotropy model was implemented and applied to the full 2D version of UEDGE including recycled neutrals. The simulation results showed that the mirror force due to anisotropy reduces ion parallel flow velocity, however, the simulations were presented without cross‐field electric and magnetic drifts. Nevertheless, in present medium‐size tokamaks or future compact fusion devices, cross‐fields are important. In this work, the anisotropy model is extended to be compatible with the inclusion of cross‐field drifts. Preliminary results on comparison between the model solutions with and without cross‐field drifts are presented.

Zhao, Menglong↗
DOE OSTI2024

Enabling Parallel Performance and Portability of Solid Mechanics Simulations Across CPU and GPU Architectures

Efficiently simulating solid mechanics is vital across various engineering applications. As constitutive models grow more complex and simulations scale up in size, harnessing the capabilities of modern computer architectures has become essential for achieving timely results. This paper presents advancements in running parallel simulations of solid mechanics on multi-core CPUs and GPUs using a single-code implementation. This portability is made possible by the C++ matrix and array (MATAR) library, which interfaces with the C++ Kokkos library, enabling the selection of fine-grained parallelism backends (e.g., CUDA, HIP, OpenMP, pthreads, etc.) at compile time. MATAR simplifies the transition from Fortran to C++ and Kokkos, making it easier to modernize legacy solid mechanics codes. We applied this approach to modernize a suite of constitutive models and to demonstrate substantial performance improvements across different computer architectures. This paper includes comparative performance studies using multi-core CPUs along with AMD and NVIDIA GPUs. Results are presented using a hypoelastic–plastic model, a crystal plasticity model, and the viscoplastic self-consistent generalized material model (VPSC-GMM). The results underscore the potential of using the MATAR library and modern computer architectures to accelerate solid mechanics simulations.

Morgan, Nathaniel (ORCID:0000000276118449)↗
DOE OSTI2021

Algebraic Algorithms for Betweenness and Percolation Centrality

Abstract In this paper, we explored different ways to write the algebraic version of betweenness centrality algorithm. Particularly, we focused on Brandes' algorithm. We aimed for algebraic betweenness centrality that can be parallelized easily. We proposed 3-tuple geodetic semiring as an extension to the usual geodetic semiring with 2-tuples. Using the 3-tuple geodetic semiring, Dijkstra's and Brandes' algorithm, we wrote more concise and general algebraic betweenness centrality (ABC) algorithm which is valid for weighted and directed graphs. We also proposed an alternative version of ABC using the usual geodetic semiring with 2-tuple where we used a simple way to construct shortest path tree after computing shortest path distances in the usual geodetic semiring. This allows us to avoid computational complexity of ABC implementation using 3-tuple geodetic semiring. We used numba to optimize and parallelize ABC. We evaluated the performance of ABC using 2-tuple geodetic semiring as compared to NetworkX, a common python package for graph algorithms. We did scalability experiments on parallel ABC and showed its total speedup. We also showed that with small modification, ABC can be adapted to algebraicly compute other centrality measures such as percolation centrality.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

A practical guide to replica-exchange Wang—Landau simulations

This paper is based on a series of tutorial lectures about the replica-exchange Wang-Landau (REWL) method given at the IX Brazilian Meeting on Simulational Physics (BMSP 2017). It provides a practical guide for the implementation of the method. A complete example code for a model system is available online. In this paper, we discuss the main parallel features of this code after a brief introduction to the REWL algorithm. The tutorial section is mainly directed at users who have written a single-walker Wang–Landau program already but might have just taken their first steps in parallel programming using the Message Passing Interface (MPI). In the last section, we answer "frequently asked questions" from users about the implementation of REWL for different scientific problems.

Vogel, Thomas↗
DOE OSTI2025

Modified Andronov-Hopf Oscillator-Based Grid-Forming Converter with Emulated Virtual Cable for Enhanced Power Sharing Performance

Nonlinear oscillator-based grid-forming converters offer superior dynamic and steady-state performance, making them an attractive solution for interconnecting renewable resources. This paper proposes a novel modified Andronov-Hopf oscillator to enhance the operating spectrum and facilitate the integration of renewable energy sources. An inner loop controller based on the Lyapunov energy function is implemented to achieve robust stability and performance, while a virtual cable emulation strategy enables seamless parallel operation. Comprehensive modeling and simulation studies validate the effectiveness of the proposed system, demonstrating its capabilities in addressing diverse operating scenarios, including grid faults, renewable energy fluctuations, and parallel operation. The proposed solution exhibits fast transient response, robust stability, and flexible operation, making it a valuable contribution to the field of renewable energy integration. The results of this study can be used to inform the design and implementation of next-generation grid-forming converters, enabling a more sustainable and reliable energy future. Additionally, the proposed system's ability to operate in both grid-connected and islanded modes makes it an ideal candidate for remote and off-grid renewable energy applications. The proposed solution's scalability and modularity also make it suitable for large-scale renewable energy integration. The proposed system is verified through MATLAB/Simulink and PLECS simulations, demonstrating its effectiveness in ensuring robust and efficient operation.

Andronov-Hopf Oscillator (AHO)↗
DOE OSTI2016

Collaborative Research: A Model of Partially Ionized Plasma Flows with Kinetic Treatment of Neutral Atoms and Nonthermal Ions

Interactions of flows of partially ionized, magnetized plasma are frequently accompanied by the presence of both thermal and non-thermal (pickup) ion components. Such interactions cannot be modeled using traditional MHD equations and require more advanced approaches to treat them. If a nonthermal component of ions is formed due to charge exchange and collisions between the thermal (core) ions and neutrals, it experiences the action of magnetic field, its distribution function is isotropized, and it soon acquires the velocity of the ambient plasma without being thermodynamically equilibrated. This situation, e. g., takes place in the outer heliosphere –- the part of interstellar space beyond the solar system whose properties are determined by the solar wind interaction with the local interstellar medium. This is also possible in laboratory, at million degrees and above, when plasma is conducting electricity far too well, which makes Ohmic heating ineffective. To attain the target temperatures one needs additional heating eventually playing a dominant role. Among such sources is a so-called neutral particle beam heating. This is a wide-spread technique (Joint European Torus and International Thermonuclear Experimental Reactor experiments) based on the injection of powerful beams of neutral atoms into ohmically preheated plasma. In this project we have investigated the energy and density separation between the thermal and nonthermal components in the solar wind and interstellar plasmas. A new model has been developed in which we solve the ideal MHD equations for mixture of all ions and the kinetic Boltzmann equation to describe the transport of neutral atoms. As a separate capability, we can treat the flow of neutral atoms in a multi-component fashion, where neutral atoms born in each thermodynamically distinct region are governed by the Euler gas dynamic equations. We also describe the behavior of pickup ions either kinetically, using the Fokker--Planck equation, or as a separate fluid. Our numerical simulations have demonstrated that pickup ions play a major role in the interaction of the solar wind and (partially ionized) interstellar medium plasmas. Our teams have investigated the stability of the surface (the heliopause) that separates the solar wind from the local interstellar medium, the transport of galactic cosmic rays, the properties of the heliotail flow, and modifications to the bow wave in front of the heliopause due to charge exchange between the neutral H atoms born in the solar wind and interstellar ions. Modeling results have been validated against observational data, such as obtained by the Interstellar Boundary Explorer (IBEX), and made it possible to shed light on the structure of energetic neutral atom maps created by this spacecraft.. We have also demonstrated that charge-exchange modulated heliosphere is a source of anisotropy of the multi-TeV cosmic ray flux observed in a number of Earth-bound air shower experiments. Newly developed codes are implemented within a Multi-Scale Fluid-Kinetic Simulation Suite (MS-FLUKSS), a publicly available code being developed by our team for over 12 years. MS-FLUKSS scales well up to 160,000 computing cores and has been ported on major supercomputers in the country. Efficient parallelization and data choreography in the continuum simulation modules are provided by Chombo, an adaptive mesh refinement framework managed by Phillip Colella's team at LBNL. We have implemented in-house, hybrid (MPI+OpenMP) parallelization of the kinetic modules that solve the Boltzmann equation with a Monte Carlo method. Currently, the kinetic modules are being rewritten to take advantage of the modern CPU-GPU supercomputer architecture. The scope of the project allowed us to enhance plasma research and education in such broad, multidisciplinary field as physics of partially ionized plasma and its application to space physics and fusion science. Besides the impact on the modeling of complex physical systems, our approach to computational resource management for complex codes utilizing multiple algorithm technologies appears to be a major advance on current approaches. The development of sophisticated resource management will be essential for all future modeling efforts that incorporate a diversity of scales and physical processes. Our effort provided leadership in promoting computational science and plasma physics within the UAH and FIT campuses and, through the training of a broad spectrum of scientists and engineers, fostering new technologies across the country.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

Collaborative Research: A Model of Partially Ionized Plasma Flows with Kinetic Treatment of Neutral Atoms and Nonthermal Ions

Interactions of flows of partially ionized, magnetized plasma are frequently accompanied by the presence of both thermal and non-thermal (pickup) ion components. Such interactions cannot be modeled using traditional MHD equations and require more advanced approaches to treat them. If a nonthermal component of ions is formed due to charge exchange and collisions between the thermal (core) ions and neutrals, it experiences the action of magnetic field, its distribution function is isotropized, and it soon acquires the velocity of the ambient plasma without being thermodynamically equilibrated. This situation, e. g., takes place in the outer heliosphere - the part of interstellar space beyond the solar system whose properties are determined by the solar wind interaction with the local interstellar medium. This is also possible in laboratory, at million degrees and above, when plasma is conducting electricity far too well, which makes Ohmic heating ineffective. To attain the target temperatures one needs additional heating eventually playing a dominant role. Among such sources is a so-called neutral particle beam heating. This is a wide-spread technique (Joint European Torus and International Thermonuclear Experimental Reactor experiments) based on the injection of powerful beams of neutral atoms into ohmically preheated plasma. In this project we have investigated the energy and density separation between the thermal and nonthermal components in the solar wind and interstellar plasmas. A new model has been developed in which we solve the ideal MHD equations for mixture of all ions and the kinetic Boltzmann equation to describe the transport of neutral atoms. As a separate capability, we can treat the flow of neutral atoms in a multi-component fashion, where neutral atoms born in each thermodynamically distinct regions are governed by the Euler gas dynamic equations. We also describe the behavior of pickup ions either kinetically, using the Fokker–Planck equation, or as a separate fluid. Our numerical simulations have demonstrated that pickup ions play a major role in the interaction of the solar wind and (partially ionized) interstellar medium plasmas. Our teams have investigated the stability of the surface (the heliopause) that separates the solar wind from the local interstellar medium, the transport of galactic cosmic rays, the properties of the heliotail flow, and modifications to the bow wave in front of the heliopause due to charge exchange between the neutral H atoms born in the solar wind and interstellar ions. Modeling results have been validated against observational data, such as obtained by the Interstellar Boundary Explorer (IBEX), and made it possible to shed light on the structure of energetic neutral atom maps created by this spacecraft.. We have also demonstrated that charge-exchange modulated heliosphere is a source of anisotropy of the multi-TeV cosmic ray flux observed in a number of Earth-bound air shower experiments. Newly developed codes are implemented within a Multi-Scale Fluid-Kinetic Simulation Suite (MS-FLUKSS), a publicly available code being developed by our team for over 12 years. MS-FLUKSS scales well up to 160,000 computing cores and has been ported on major supercomputers in the country. Efficient parallelization and data choreography in the continuum simulation modules are provided by Chombo, an adaptive mesh refinement framework managed by Phillip Colella’s team at LBNL. We have implemented in-house, hybrid (MPI+OpenMP) parallelization of the kinetic modules that solve the Boltzmann equation with a Monte Carlo method. Currently, the kinetic modules are being rewritten to take advantage of the modern CPU-GPU supercomputer architecture. The scope of the project allowed us to enhance plasma research and education in such broad, multidis- ciplinary field as physics of partially ionized plasma and its application to space physics and fusion science. Besides the impact on the modeling of complex physical systems, our approach to computational resource management for complex codes utilizing multiple algorithm technologies appears to be a major advance on current approaches. The development of sophisticated resource management will be essential for all future modeling efforts that incorporate a diversity of scales and physical processes. Our effort provided leadership in promoting computational science and plasma physics within the UAH and FIT campuses and, through the training of a broad spectrum of scientists and engineers, foster new technologies across the country.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Community Detection on the GPU

We present and evaluate a new GPU algorithm based on the Louvain method for community detection. Our algorithm is the first for this problem that parallelizes the access to individual edges. In this way we can fine tune the load balance when processing networks with nodes of highly varying degrees. This is achieved by scaling the number of threads assigned to each node according to its degree. Extensive experiments show that we obtain speedups up to a factor of 270 compared to the sequential algorithm. The algorithm consistently outperforms other recent shared memory implementations and is only one order of magnitude slower than the current fastest parallel Louvain method running on a Blue Gene/Q supercomputer using more than 500K threads.

community detection↗
DOE OSTI2017

Tuning HDF5 subfiling performance on parallel file systems

Subfiling is a technique used on parallel file systems to reduce locking and contention issues when multiple compute nodes interact with the same storage target node. Subfiling provides a compromise between the single shared file approach that instigates the lock contention problems on parallel file systems and having one file per process, which results in generating a massive and unmanageable number of files. In this paper, we evaluate and tune the performance of recently implemented subfiling feature in HDF5. In specific, we explain the implementation strategy of subfiling feature in HDF5, provide examples of using the feature, and evaluate and tune parallel I/O performance of this feature with parallel file systems of the Cray XC40 system at NERSC (Cori) that include a burst buffer storage and a Lustre disk-based storage. We also evaluate I/O performance on the Cray XC30 system, Edison, at NERSC. Our results show performance benefits of 1.2X to 6X performance advantage with subfiling compared to writing a single shared HDF5 file. We present our exploration of configurations, such as the number of subfiles and the number of Lustre storage targets to storing files, as optimization parameters to obtain superior I/O performance. Based on this exploration, we discuss recommendations for achieving good I/O performance as well as limitations with using the subfiling feature.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

An adaptive scalable fully implicit algorithm based on stabilized finite element for reduced visco-resistive MHD

The magnetohydrodynamics (MHD) equations are continuum models used in the study of a wide range of plasma physics systems, including the evolution of complex plasma dynamics in tokamak disruptions. However, efficient numerical solution methods for MHD are extremely challenging due to disparate time and length scales, strong hyperbolic phenomena, and nonlinearity. Additionally, therefore the development of scalable, implicit MHD algorithms and high-resolution adaptive mesh refinement strategies is of considerable importance. In this work, we develop a high-order stabilized finite-element algorithm for the reduced visco-resistive MHD equations based on the MFEM finite element library (mfem.org). The scheme is fully implicit, solved with the Jacobian-free Newton-Krylov (JFNK) method with a physics-based preconditioning strategy. Our preconditioning strategy is a generalization of the physics-based preconditioning methods in Chacón et al. (2002) to adaptive, stabilized finite elements. Algebraic multigrid methods are used to invert sub-block operators to achieve scalability. A parallel adaptive mesh refinement scheme with dynamic load-balancing is implemented to efficiently resolve the multi-scale spatial features of the system. Our implementation uses the MFEM framework, which provides arbitrary-order polynomials and flexible adaptive conforming and non-conforming meshes capabilities. Results demonstrate the accuracy, efficiency, and scalability of the implicit scheme in the presence of large scale disparity. The potential of the AMR approach is demonstrated on an island coalescence problem in the high Lundquist-number regime (≥ 10 7 ) with the successful resolution of plasmoid instabilities and thin current sheets.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2016

Petascale Orbital-Free Density Functional Theory Enabled by Small-Box Algorithms

Orbital-free density functional theory (OFDFT) is a quantum-mechanics-based method that utilizes electron density as its sole variable. The main computational cost in OFDFT is the ubiquitous use of the fast Fourier transform (FFT), which is mainly adopted to evaluate the kinetic energy density functional (KEDF) and electron–electron Coulomb interaction terms. We design and implement a small-box FFT (SBFFT) algorithm to overcome the parallelization limitations of conventional FFT algorithms. We also propose real-space truncation of the nonlocal Wang–Teter KEDF kernel. The scalability of the SBFFT is demonstrated by efficiently simulating one full optimization step (electron density, energies, forces, and stresses) of 1,024,000 lithium (Li) atoms on up to 65,536 cores. We perform other tests using Li as a test material, including calculations of physical properties of different phases of bulk Li, geometry optimizations of nanocrystalline Li, and molecular dynamics simulations of liquid Li. All of the tests yield excellent agreement with the original OFDFT results, suggesting that the OFDFT-SBFFT algorithm opens the door to efficient first-principles simulations of materials containing millions of atoms.

Chemistry↗
DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗