Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed parallelization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25

A nested dissection approach to sparse matrix partitioning for parallel computations.

We consider how to distribute sparse matrices among processes to reduce communication costs in parallel sparse matrix computations, specifically, sparse matrix-vector multiplication. Our main contributions are: (i) an exact graph model for communication with general (two-dimensional) matrix distribution, and (ii) a recursive partitioning algorithm based on nested dissection (substructuring). We show that the communication volume is closely linked to vertex separators. We have implemented our algorithm using hypergraph partitioning software to enable a fair comparison with existing methods. We present numerical results for sparse matrices from several application areas, with up to 9 million nonzeros. The results show that our new approach is superior to traditional 1d partitioning and comparable to a current leading partitioning method, the finegrain hypergraph method, in terms of communication volume. Our nested dissection method has two advantages over the fine-grain method: it is faster to compute, and the resulting distribution requires fewer communication messages.

97 MATHEMATICS AND COMPUTING↗

PREEMPT: Scalable Epidemic Interventions Using Submodular Optimization on Multi-GPU Systems

Preventing and slowing the spread of epidemics is achieved through techniques such as vaccination and social distancing. Given practical limitations on the number of vaccines and cost of administration, optimization becomes a necessity. Previous approaches using mathematical programming methods have shown to be effective but are limited by computational costs. In this work, we make several contributions: First, we present a new approach for intervention via maximizing the influence of vaccinated nodes on the network. We call this method \preempt. Next, we prove submodular properties associated with the objective function of our method so that it aids in construction of an efficient greedy approximation strategy. Consequently, we present a new parallel algorithm based on greedy hill climbing for \preempt, and present an efficient parallel implementation for distributed CPU-GPU heterogeneous platforms. Our results demonstrate that \preempt{} is able to achieve a significant reduction (up to 6.75$\times$) in the percentage of people infected on a city-scale network. We also show strong scaling results of \preempt{} on 128 nodes of the Summit supercomputer. Our parallel implementation is able to significantly reduce time to solution, from hours to minutes on large networks. This work represents a first-of-its-kind effort in parallelizing greedy hill climbing and applying it toward devising effective interventions for epidemics.

Minutoli, Marco↗

DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.

checkpointing↗

Using Distributed Energy Resources to Supply Reactive Power for Dynamic Voltage Regulation

Abstract Distributed energy (DE) resources are power sources located near load centers and equipped with power electronics converters to interface with the grid, therefore it is feasible for DE to provide reactive power (along with active power) locally for dynamic voltage regulation. In this paper, a synchronous condenser and a DE source with an inverter interface are implemented in parallel in a distribution system to regulate the local voltage. Developed voltage control schemes for the inverter and the synchronous condenser are presented. Experimental results show that both the inverter and the synchronous condenser can regulate the local voltage instantaneously although the dynamic response of the inverter is much faster than the synchronous condenser. In a system with multiple DEs performing local voltage regulation, the interaction of multiple DE at different locations under different load levels may have an impact to the control parameter setting for each individual DE control system. Future research is needed to find out the interaction of DEs to identify the optimal control parameter settings with the consideration of many factors such as system configuration, load variation, and so on

24 POWER TRANSMISSION AND DISTRIBUTION↗

Analysis techniques for diagnosing runaway ion distributions in the reversed field pinch

An advanced neutral particle analyzer (ANPA) on the Madison Symmetric Torus measures deuterium ions of energy ranges 8-45 keV with an energy resolution of 2-4 keV and time resolution of 10 μs. Three different experimental configurations measure distinct portions of the naturally occurring fast ion distributions: fast ions moving parallel, anti-parallel, or perpendicular to the plasma current. On a radial-facing port, fast ions moving perpendicular to the current have the necessary pitch to be measured by the ANPA. With the diagnostic positioned on a tangent line through the plasma core, a chord integration over fast ion density, background neutral density, and local appropriate pitch defines the measured sample. The plasma current can be reversed to measure anti-parallel fast ions in the same configuration. Comparisons of energy distributions for the three configurations show an anisotropic fast ion distribution favoring high pitch ions.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

GronOR: Massively Parallel and GPU-Accelerated Non-Orthogonal Configuration Interaction for Large Molecular Systems

GronOR is a program package for non-orthogonal configuration interaction calculations for an electronic wave function built in terms of anti-symmetrized products of multi-configuration molecular fragment wave functions. The two-electron integrals that have to be processed may be expressed in terms of atomic orbitals or in terms of an orbital basis determined from the molecular orbitals of the fragments. The code has been specifically designed for execution on distributed memory massively parallel and Graphics Processing Unit (GPU)-accelerated computer architectures, using an MPI+OpenACC/OpenMP programming approach. The task-based execution model used in the implementation allows for linear scaling with the number of nodes on the largest pre-exascale architectures available, provides hardware fault resiliency, and enables effective execution on systems with distinct central processing unit-only and GPU-accelerated partitions. The code interfaces with existing multi-configuration electronic structure codes that provide optimized molecular fragment orbitals, configuration interaction coefficients, and the required integrals. Algorithm and implementation details, parallel and accelerated performance benchmarks, and an analysis of the sensitivity of the accuracy of results and computational performance to thresholds used in the calculations are presented.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Optimization and Parallelization of the Thermal-Hydraulic Sub-channel Code CTF for High-Fidelity Multi-physics Applications

This paper describes major improvements to the computational infrastructure of the CTF sub-channel code so that full-core sub-channel-resolved simulations can now be performed in much shorter run-times, either in stand-alone mode or as part of coupled-code multi-physics calculations. These improvements support the goals of the Department Of Energy (DOE) Consortium for Advanced Simulations of Light Water (CASL) Energy Innovation Hub to develop high fidelity multi-physics simulation tools for nuclear energy design and analysis. A set of serial code optimizations--including fixing computational inefficiencies, optimizing the numerical approach, and making smarter data storage choices--are first described and shown to reduce both execution time and memory usage by about a factor of ten. Next, a Single Program Multiple Data (SPMD) parallelization strategy targeting distributed memory Multiple Instruction Multiple Data (MIMD) platforms and utilizing domain-decomposition is presented. In this approach, data communication between processors is accomplished by inserting standard MPI calls at strategic points in the code. The domain decomposition approach implemented assigns one MPI process to each fuel assembly, with each domain being represented by its own CTF input file. The creation of CTF input files, both for serial and parallel runs, is also fully automated through use of a pre-processor utility that takes a greatly reduced set of user input over the traditional CTF input file. To run CTF in parallel, two additional libraries are currently needed; MPI, for inter-processor message passing, and the Parallel Extensible Toolkit for Scientific Computation (PETSc), which is leveraged to solve the global pressure matrix in parallel. Results presented include a set of testing and verification calculations and performance tests assessing parallel scaling characteristics up to a full core, sub-channel-resolved model of Watts Bar Unit 1 under hot full-power conditions (193 17x17 assemblies, ~56,000 pins, ~59,000 sub-channels, ~2.8 million thermal-hydraulic (TH) control volumes). Results demonstrate that CTF can now perform full core analysis (not previously possible due to excessively long runtimes and memory requirements) on the order of 20 minutes. This new capability is not only useful to standalone CTF users, but is also being leveraged in support of coupled code multi-physics calculations being done in the CASL program.

COBRA-TF↗

Global Arrays Parallel Programming Toolkit

The two predominant classes of programming models for parallel computing are distributed memory and shared memory. Both shared memory and distributed memory models have advantages and shortcomings. Shared memory model is much easier to use but it ignores data locality/placement. Given the hierarchical nature of the memory subsystems in modern computers this characteristic can have a negative impact on performance and scalability. Careful code restructuring to increase data reuse and replacing fine grain load/stores with block access to shared data can address the problem and yield performance for shared memory that is competitive with message-passing. However, this performance comes at the cost of compromising the ease of use that the shared memory model advertises. Distributed memory models, such as message-passing or one-sided communication, offer performance and scalability but they are difficult to program. The Global Arrays toolkit attempts to offer the best features of both models. It implements a shared-memory programming model in which data locality is managed by the programmer. This management is achieved by calls to functions that transfer data between a global address space (a distributed array) and local storage. In this respect, the GA model has similarities to the distributed shared-memory models that provide an explicit acquire/release protocol. However, the GA model acknowledges that remote data is slower to access than local data and allows data locality to be specified by the programmer and hence managed. GA is related to the global address space languages such as UPC, Titanium, and, to a lesser extent, Co-Array Fortran. In addition, by providing a set of data-parallel operations, GA is also related to data-parallel languages such as HPF, ZPL, and Data Parallel C. However, the Global Array programming model is implemented as a library that works with most languages used for technical computing and does not rely on compiler technology for achieving parallel efficiency. It also supports a combination of task- and data-parallelism and is available as an extension of the message passing (MPI) model. The GA model exposes to the programmer the hierarchical memory of modern high-performance computer systems, and by recognizing the communication overhead for remote data transfer, it promotes data reuse and locality of reference. Virtually all the scalable architectures possess non-uniform memory access characteristics that reflect their multi-level memory hierarchies. These hierarchies typically comprise processor registers, multiple levels of cache, local memory, and remote memory. Over time, both the number of levels and the cost (in processor cycles) of accessing deeper levels has been increasing. It is important for any scalable programming model to address memory hierarchy since it is critical to the efficient execution of scalable applications.

ARMCI↗

Electron acceleration by magnetic islands in a dynamically evolved coronal current sheet

This work simulated the electron acceleration by magnetic islands in a drastically evolved solar coronal current sheet via the combined 2.5-dimensional (2.5D) resistive Magnetohydrodynamics (MHD) and guiding-center approximation test-particle methods. With high magnetic Reynolds number of 105, the long–thin current sheet is evolved into a chain of magnetic islands, growing in size and coalescing with each other, due to tearing instability. The acceleration of electrons is studied in one typical phase when several large magnetic islands are formed. The results show that the electrons with an initial Maxwell distribution evolve into a heavy-tailed distribution and more than 20% of the electrons can be accelerated higher than 200 keV within 0.1 second and some of them can even be energized up to MeV ranges. The most energetic electrons have a tendency to be around the outer regions of the magnetic islands or to be located in the small secondary magnetic islands. We find that the acceleration and spatial distributions of the energetic electrons is caused by the trapping effect of the magnetic islands and the distributions of the parallel electric field E{sub p}.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Tomographic analysis of tangential viewing cameras (invited)

Many tokamaks now use visible light cameras to observe plasma-wall interactions and integrated line emission. The DIII-D coherence imaging spectroscopy diagnostic cameras image interferograms that encode line integrated velocity. By modeling the 2D camera image pixels as line of sight integrals through an axisymmetric discrete grid, it is possible to do tomographic analysis to determine the local plasma line emissivity and parallel velocity. Approaches to solve the inverse problem posed by these tangential viewing cameras are presented. The inversion begins with calculation of the sparse response matrix that encompasses all the geometry and diagnostic information and reduces the process of image formation to a sparse matrix-vector multiply. Our report includes techniques for determining the detailed geometry of the camera views and methods for handling physical quantities that vary spatially. Additionally, the size of the response matrix has driven the development of capability to distribute the coarse parallel calculation across a heterogeneous cluster of computers on the Energy Sciences Network. Iterative techniques are then used to solve the sparse matrix-vector linear system

47 OTHER INSTRUMENTATION↗

Ion-cyclotron instability in plasmas described by product-bi-kappa distributions

The dispersion relation for parallel propagating waves in the ion-cyclotron branch is investigated numerically by considering that the velocity distribution of the ion population is a function of type product-bi-kappa. We investigate the effects of the non-thermal features and of the anisotropy associated with this type of distribution on the ion-cyclotron instability, as well as the influence of different forms of the electron distribution, by considering Maxwellian distributions, bi-kappa distributions, and product-bi-kappa distributions. The cases of ions described by either Maxwellian or bi-kappa distributions are also considered, for comparison. The results of the numerical analysis show that the increase in the non-thermal character associated with the anisotropic kappa distributions for ions contributes to enhance the instability as compared to that obtained in the Maxwellian case, in magnitude and in wave number range, with more significant enhancement for the case of ion product-bi-kappa distributions than for the case of ion bi-kappa distributions. It is also shown that the ion-cyclotron instability is decreased if the electrons are described by product-bi-kappa distributions, while electrons described by bi-kappa distributions lead to growth rates which are very similar to those obtained considering a Maxwellian distribution for the electron population.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

A new sawtooth control mechanism relying on toroidally propagating ion cyclotron resonance frequency waves: Theory and Joint European Torus tokamak experimental evidence

The sawtooth control mechanism in plasmas employing toroidally propagating ion cyclotron resonance waves is extended. The asymmetrically distributed energetic passing ions are shown to modify the ideal internal kink mode when the position of the minority ion cyclotron resonance resides within a narrow region close to the q=1 surface. An analytical treatment of the internal kink mode in the presence of model distribution function with parallel velocity asymmetry is developed. The fast ion mechanism explains the strong sensitivity of sawteeth to resonance position, and moreover is consistent with dedicated Joint European Torus [F. Romanelli, Nucl. Fusion 49, 104006 (2009)] experiments which controlled sawteeth despite negligible current drive.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

A Game-Theoretic Data-Driven Approach for Pseudo-Measurement Generation in Distribution System State Estimation

In this paper, we present an efficient computational framework with the purpose of generating weighted pseudo-measurements to improve the quality of distribution system state estimation (DSSE) and provide observability with advanced metering infrastructure (AMI) against unobservable customers and missing data. The proposed technique is based on a game-theoretic expansion of relevance vector machines (RVMs). This platform is able to estimate the nodal power consumption and quantify its uncertainty while reducing the prohibitive computational burden of model training for large AMI datasets. To achieve this objective, the large training set is decomposed and distributed among multiple parallel learning entities. The resulting estimations from the parallel RVMs are then combined using a game-theoretic model based on the idea of repeated games with vector payoff. It is observed that through this approach and by exploiting the seasonal changes in customers’ behavior the accuracy of pseudo-measurements can be considerably improved, while introducing robustness against bad training data samples. The proposed pseudo-measurement generation model is integrated into a DSSE using a closed-loop information system, which takes advantage of a branch current state estimator (BCSE) to further improve the performance of the designed machine learning framework. This method has been tested on a practical distribution feeder model with smart meter data for verification.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Tomosaic : efficient acquisition and reconstruction of teravoxel tomography data using limited-size synchrotron X-ray beams

X-rays offer high penetration with the potential for tomography of centimetre-sized specimens, but synchrotron beamlines often provide illumination that is only millimetres wide. Here an approach is demonstrated termed Tomosaic for tomographic imaging of large samples that extend beyond the illumination field of view of an X-ray imaging system. This includes software modules for image stitching and calibration, while making use of existing modules available in other packages for alignment and reconstruction. The approach is compatible with conventional beamline hardware, while providing a dose-efficient method of data acquisition. By using parallelization on a distributed computing system, it provides a solution for handling teravoxel-sized or larger datasets that cannot be processed on a single workstation in a reasonable time. Using experimental data, the package is shown to provide good quality three-dimensional reconstruction for centimetre-sized samples with sub-micrometre pixel size.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Dense LU Factorization on Multicore Supercomputer Nodes

Dense LU factorization is a prominent benchmark used to rank the performance of supercomputers. Many implementations, including the reference code HPL, use block-cyclic distributions of matrix blocks onto a two-dimensional process grid. The process grid dimensions drive a trade-off between communication and computation and are architecture- and implementation-sensitive. We show how the critical panel factorization steps can be made less communication-bound by overlapping asynchronous collectives for pivot identification and exchange with the computation of rank-k updates. By shifting this trade-off, a modified block-cyclic distribution can beneficially exploit more available parallelism on the critical path, and reduce panel factorization's memory hierarchy contention on now-ubiquitous multi-core architectures. The missed parallelism in traditional block-cyclic distributions arises because active panel factorization, triangular solves, and subsequent broadcasts are spread over single process columns or rows (respectively) of the process grid. Increasing one dimension of the process grid decreases the number of distinct processes in the other dimension. To increase parallelism in both dimensions, periodic 'rotation' is applied to the process grid to recover the row-parallelism lost by a tall process grid. During active panel factorization, rank-1 updates stream through memory with minimal reuse. In a column-major process grid, the performance of this access pattern degrades as too many streaming processors contend for access to memory. A block-cyclic mapping in the more popular row-major order does not encounter this problem, but consequently sacrifices node and network locality in the critical pivoting steps. We introduce 'striding' to vary between the two extremes of row- and column-major process grids. As a test-bed for further mapping experiments, we describe a dense LU implementation that allows a block distribution to be defined as a general function of block to processor. Other mappings can be tested with only small, local changes to the code.

99 GENERAL AND MISCELLANEOUS↗

Block-Parallel Data Analysis with DIY2

DIY2 is a programming model and runtime for block-parallel analytics on distributed-memory machines. Its main abstraction is block-structured data parallelism: data are decomposed into blocks; blocks are assigned to processing elements (processes or threads); computation is described as iterations over these blocks, and communication between blocks is defined by reusable patterns. By expressing computation in this general form, the DIY2 runtime is free to optimize the movement of blocks between slow and fast memories (disk and flash vs. DRAM) and to concurrently execute blocks residing in memory with multiple threads. This enables the same program to execute in-core, out-of-core, serial, parallel, single-threaded, multithreaded, or combinations thereof. This paper describes the implementation of the main features of the DIY2 programming model and optimizations to improve performance. DIY2 is evaluated on benchmark test cases to establish baseline performance for several common patterns and on larger complete analysis codes running on large-scale HPC machines.

97 MATHEMATICS AND COMPUTING↗

Exploring Anomalous Photoelectron Angular Distributions in the Photoelectron Spectra of Gd 3 O 3 – : Study of Gd 3 O 2 – and Gd 3 O 3 – Using Photoelectron Spectroscopy and Density Functional Theory Calculations

Anion photoelectron (PE) spectra of lanthanide oxide clusters obtained previously have exhibited anomalous photoelectron angular distributions which were attributed to strong PE–valence electron (PEVE) interactions. Here, to further explore this effect, we have obtained the PE spectra of Gd 3 O 2 – and Gd 3 O 3 – , two clusters that have similarly complex electronic structures but contrasting symmetries. The spectra exhibit manifolds of detachment transitions at similar binding energies in a 0.5 eV window of energy. The electron affinity of Gd 3 O 2 is measured to be 1.29 ± 0.05 eV, and that of Gd 3 O 3 is 1.31 ± 0.05 eV. As seen in previous studies on lanthanide oxide cluster anions in lower than conventional oxidation states, transitions in spectra obtained lower photon energies are more congested than those obtained with higher photon energy, a signature of strong PEVE interactions. While the detachment transitions have predominantly parallel photoelectron angular distributions (PAD), the PAD varies across the manifold of transitions in the PE spectrum of Gd 3 O 3 – in a way that suggests four different subgroups of transitions. Results of calculations on Gd 3 O 2 – suggest kite or V-shape structures with antiferromagnetic coupling between one of the 4f 7 subshells with the two others. Calculations on Gd 3 O 3 – more definitively point to ring structures with a nearly isoenergetic ferromagnetically coupled high spin (24-tet) state and a dectet state in which one of the 4f 7 subshells is antiferromagnetically coupled with the other two. Taking these results as qualitative, we propose that strong mixing between the unperturbed states predicted computationally leads to overlapping transitions with different PADs.

anions↗