Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Understanding scale-dependent soft-error behavior of scientific applications

Analyzing application fault behavior on large-scale systems is time-consuming and resource-demanding. Currently, researchers need to perform fault injection campaigns at full scale to understand the effects of soft errors on applications and whether these faults result in silent data corruption. Both time and resource requirements greatly limit the scope of the resilience studies that can be currently performed. In this work, we propose a methodology to model application fault behavior at large scale based on a reduced set of experiments performed at small scale. We employ machine learning techniques to accurately model application fault behavior using a set of experiments that can be executed in parallel at small scale. Our methodology drastically reduces the set and the scale of the fault injection experiments to be performed and provides a validated methodology to study application fault behavior at large scale. We show that our methodology can accurately model application fault behavior at large scale by using only small scale experiments. In some cases, we can model the fault behavior of a parallel application running on 4,096 cores with about 90% accuracy based on experiments on a single core.

Kestor, Gokcen G.↗
DOE OSTI2020

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

Here, we present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up to 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Low-synch Gram–Schmidt with delayed reorthogonalization for Krylov solvers

The parallel strong-scaling of iterative methods is often determined by the number of global reductions at each iteration. Low-synch Gram-Schmidt algorithms are applied here to the Arnoldi algorithm to reduce the number of global reductions and therefore to improve the parallel strong-scaling of iterative solvers for nonsymmetric matrices such as the GMRES and the Krylov-Schur iterative methods. In the Arnoldi context, the factorization is "left-looking" and processes one column at a time. Among the methods for generating an orthogonal basis for the Arnoldi algorithm, the classical Gram-Schmidt algorithm, with reorthogonalization (CGS2) requires three global reductions per iteration. A new variant of CGS2 that requires only one reduction per iteration is presented and applied to the Arnoldi algorithm. Delayed CGS2 (DCGS2) employs the minimum number of global reductions per iteration (one) for a one-column at-a-time algorithm. The main idea behind the new algorithm is to group global reductions by rearranging the order of operations. DCGS2 must be carefully integrated into an Arnoldi expansion or a GMRES solver. Numerical stability experiments assess robustness for Krylov-Schur eigenvalue computations. Performance experiments on the ORNL Summit supercomputer then establish the superiority of DCGS2 over CGS2.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

DFT-FE – A massively parallel adaptive finite-element code for large-scale density functional theory calculations

We present an accurate, efficient and massively parallel finite-element code, DFT-FE, for large-scale ab-initio calculations (reaching ~100,000 electrons) using Kohn-Sham density functional theory (DFT). DFT-FE is based on a local real-space variational formulation of the Kohn-Sham DFT energy functional that is discretized using a higher-order adaptive spectral finite-element (FE) basis, and treats pseudopotential and all-electron calculations in the same framework, while accommodating non-periodic, semi-periodic and periodic boundary conditions. We discuss the main aspects of the code, which include, the various strategies of adaptive FE basis generation, and the different approaches employed in the numerical implementation of the solution of the discrete Kohn-Sham problem that are focused on significantly reducing the floating point operations, communication costs and latency. We demonstrate the accuracy of DFT-FE by comparing the energies, ionic forces and periodic cell stresses on a wide range of problems with popularly used DFT codes. Further, we demonstrate that DFT-FE significantly outperforms widely used plane-wave codes---both in CPU-times and wall-times, and on both non-periodic and periodic systems---at systems sizes beyond a few thousand electrons, with over 5-10 fold speedups in systems with more than 10,000 electrons. In conclusion, the benchmark studies also highlight the excellent parallel scalability of DFT-FE, with strong scaling demonstrated on up to 192,000 MPI tasks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A Robust Parallel Distributed State Estimation for Large Scale Distribution Systems

The growing need and interest in real-time monitoring of large distribution networks motivated by the rapid population of renewable sources, EVs and etc. demand a computationally efficient state estimation framework. Furthermore, this paper presents an improved computational framework for implementing a robust state estimator using a multi-core processor. The main contribution of the paper is the proposed computational framework along with two partitioning strategies which enable fast and robust state estimation for large scale radial and/or meshed distribution systems. Formulation of the proposed method and its implementation are described in detail. Performance of the estimator is tested by simulations first using a small 84-bus radial distribution system. Then the method’s scalability is demonstrated by simulations on two very large scale distribution networks one configured radially and the other meshed each containing over 12,500 buses.

42 ENGINEERING↗
DOE OSTI2021

Large-Scale Welding Process Simulation by GPU Parallelized Computing

The computational design of industrially relevant welded structures is extremely time consuming due to coupled physics and high nonlinearity. Previously, most welding distortion and residual stress simulations have been limited to small coupons and reduced order (from three-dimensional [3D] to two-dimensional [2D]), or inherent strain approximations were used for large structures. In this current study, an explicit finite element code based on a graphics processing unit was utilized to perform 3D transient thermomechanical simulation of structural components during welding. Laser brazing of aluminum alloy panels as representative of automotive manufacturing scenarios was simulated to predict out-of-plane distortion under different clamping conditions. The predicted deformation pattern and magnitude were validated by laser scanning data of physical assemblies. In addition, the code was used to investigate residual stresses developed during multipass arc welding of a nuclear industry pressurizer surge nozzle and subsequent welding repair where a 3D simulation was necessary. Taking the experimental data as reference, the 3D model predicted better residual stress distribution than a typical 2D asymmetrical model. Stress evolution in welding repair was also presented and discussed in this study. Furthermore, the efficient numerical model made it feasible to use integrated computational welding engineering to simulate welding processes for large-scale structures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism

Graph neural networks (GNNs), an emerging class of machine learning models for graphs, have gained popularity for their superior performance in various graph analytical tasks. Mini-batch training is commonly used to train GNNs on large graphs, and data parallelism is the standard approach to scale mini-batch training across multiple GPUs. Data parallel approaches contain redundant work as subgraphs sampled by different GPUs contain significant overlap. To address this issue, we introduce a hybrid parallel mini-batch training paradigm called Split parallelism. Split parallelism avoids redundant work by splitting the sampling, loading, and training of each mini-batch across multiple GPUs. Split parallelism, however, introduces communication overheads that can be more than the savings from removing redundant work. We further present a lightweight partitioning algorithm that probabilistically minimizes these overheads. We implement spllit parllelism in GSplit and show that it outperforms state-of-the-art mini-batch training systems like DGL, Quiver, and P3.

Lim, Seung-Hwan [ORNL] (ORCID:0000000194616866)↗
DOE OSTI2022

Parallel Multigrid in Time and Space for Extreme-Scale Computational Science: Chaotic and Hyperbolic Problems

The coming massive parallelism of exascale computing presents a pressing challenge for the many DOE simulations of time-dependent partial differential equations (PDEs), which typically use traditional sequential time stepping methods. Since this traditional approach is inherently serial, it presents a sequential bottleneck when moving to exascale computing, because future performance gains will come through greater concurrency, not faster clock speeds. Thus, the goal of this work is to research parallelism in time, i.e., methods that compute multiple time values simultaneously, not sequentially. The focus will be on hyperbolic and chaotic problems of interest to DOE, with the goal of enabling scalable simulations of time-dependent hyperbolic and chaotic problems on future architectures. The chosen methodology for solving these problems parallel-in-time is multigrid, because multigrid (when it works) is a powerful, optimal, and scalable solver for discretized PDEs. Multigrid is already commonly used in many DOE simulations for scalably and optimally solving space-only PDE problems. The areas of hyperbolic and chaotic problems are chosen because of their relevance to problems of programmatic interest to DOE. However, these problems are also well-known to be difficult for parallelin-time methods, with the most common method, parareal, diverging in many cases. The current stateof-the-art for parallel-in-time at LLNL is the multigrid reduction in time (MGRIT) XBraid package, which also struggles for such problems, while still showing some improvement over parareal. In summary, new methods are needed for an efficient parallel-in-time scheme for hyperbolic and chaotic problems, and this work shall research promising new multigrid methods in this area. In particular, we take inspiration from the Least Squares Shadowing (LSS by Wang) approach for solving chaotic problems. Here, an optimization approach is able to find “well-conditioned” shadow trajectories/solutions to the original “ill-conditioned” chaotic problem. Thus, the new multigrid methods researched here also arise in an optimization context.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

IDAES-PSE 2.7.0 Release

The Institute for the Design of Advanced Energy Systems (IDAES) Integrated Platform is a versatile computational environment offering extensive process systems engineering (PSE) capabilities for optimizing the design and operation of complex, interacting technologies and systems. IDAES enables users to efficiently search vast, complex design spaces to discover the lowest cost solutions while supporting the full process modeling lifecycle, from conceptual design to dynamic optimization and control. The extensible, open platform empowers users to create models of novel processes and rapidly develop custom analyses, workflows, and end-user applications. IDAES-PSE 2.7.0 Release Highlights New features: AutoScaler and CustomScalerBase classes: Such tools are the core of the new scaling framework being implemented in IDAES. Wider adoption of scaling tools among users will result in quicker and more robust model solutions. Scaler for equilibrium reactor and saponification properties: These scaler models are examples to follow for how to use the new scaling tools. ONNX Surrogate support from Optimization & Machine Learning Toolkit (OMLT): ONNX is an open standard format to save and load ML/AI models that is widely supported by all major frameworks. This capability makes it easier for IDAES users to create surrogate models and use them without having to support each framework individually. 1D Membrane Model for CO2 Capture and Utilization: Supports ongoing efforts for modeling and optimizing polymer membrane processes for CO2 capture and conversion into formic acid. StreamScaler unit model: Unrelated to the CustomScalerBase, this unit model allows a stream’s extensive variables to be scaled by a fixed factor. This allows streams being processed by multiple units in parallel to be scaled down to unit scale and scaled back up to process scale. Bug fixes or improvements: Scaling, EoS, Diagnostics tool, Modular Properties, tests & documentation Deprecations: Old Cubic EoS

AS↗
DOE OSTI2025

ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling

Sparse observations and coarse-resolution climate models limit effective regional decision-making, underscoring the need for robust downscaling. However, existing AI methods struggle with generalization across variables and geographies and are constrained by the quadratic complexity of Vision Transformer (ViT) self-attention. We introduce ORBIT-2, a scalable foundation model for global, hyper-resolution climate downscaling. ORBIT-2 incorporates two key innovations: (1) Residual Slim ViT (Reslim), a lightweight architecture with residual learning and Bayesian regularization for efficient, robust prediction; and (2) TILES, a tile-wise sequence scaling algorithm that reduces self-attention complexity from quadratic to linear, enabling long-sequence processing and massive parallelism. ORBIT-2 scales to 10 billion parameters across 65,536 GPUs, achieving up to 4.1 ExaFLOPS sustained throughput and 74–98% strong scaling efficiency. It supports downscaling to 0.9 km global resolution and processes sequences up to 4.2 billion tokens. On 7 km resolution benchmarks, ORBIT-2 achieves high accuracy with R2 scores in range of 0.98–0.99 against observation data.

Wang, Xiao [ORNL] (ORCID:0000000165451943)↗
DOE OSTI2025

ORBIT-2 Weather and Climate Downscaling Software Repository

ORBIT-2 is a scalable foundation model for global, hyper-resolution climate and weather downscaling. ORBIT-2 incorporates two key innovations: (1) Residual Slim ViT (Reslim), a lightweight architecture with residual learning and Bayesian regularization for efficient, robust prediction; and (2) TILES, a tile-wise sequence scaling algorithm that reduces self-attention complexity from quadratic to linear, enabling long-sequence processing and massive parallelism. ORBIT-2 scales to 10 billion parameters across 65,536 GPUs, achieving up to 4.1 ExaFLOPS sustained throughput and 74–98% strong scaling efficiency. It supports downscaling to 0.9 km global resolution and processes sequences up to 4.2 billion tokens. On 7 km resolution benchmarks, ORBIT-2 achieves high accuracy with 𝑅2 scores in range of 0.98–0.99 against observation data.

Wang, Xiao [Oak Ridge National Laboratory]↗
DOE OSTI2023

Performance Analysis and Optimal Node-aware Communication for Enlarged Conjugate Gradient Methods

Krylov methods are a key way of solving large sparse linear systems of equations but suffer from poor strong scalability on distributed memory machines. Furthermore, this is due to high synchronization costs from large numbers of collective communication calls alongside a low computational workload. Enlarged Krylov methods address this issue by decreasing the total iterations to convergence, an artifact of splitting the initial residual and resulting in operations on block vectors. In this article, we present a performance study of an enlarged Krylov method, Enlarged Conjugate Gradients (ECG), noting the impact of block vectors on parallel performance at scale. Most notably, we observe the increased overhead of point-to-point communication as a result of denser messages in the sparse matrix-block vector multiplication kernel. Additionally, we present models to analyze expected performance of ECG, as well as motivate design decisions. Most importantly, we introduce a new point-to-point communication approach based on node-aware communication techniques that increases efficiency of the method at scale.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Theory of critical balance in plasma turbulence

Critical balance, a statement of the approximate equality of a timescale of motion parallel to the magnetic field with a timescale of turbulent motion perpendicular to the field, is analyzed from a statistical description of the relative motion of two particles consistent with two-point correlation in a nonlinear gyrokinetic system with collisions. Critical balance is understood from a premise that the fastest time to any spatial scale that encounters decorrelated motions sets an overall correlation time for both perpendicular and parallel scales. For a regime of weak collisionality, this leads to the approximate equality of the parallel streaming time and the perpendicular turbulent correlation time, which is a standard statement of critical balance. These two timescales are only approximately equal because of effects in parallel streaming dynamics and collisions, the former producing deviations tied to variations of perpendicular and parallel wavenumbers. Finally, analysis of a collisional regime shows that critical balance also applies in that limit with a modified relation between parallel and perpendicular spatial scales.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

DyG-DPCD: A Distributed Parallel Community Detection Algorithm for Large-Scale Dynamic Graphs

Dynamic (Temporal) graphs capture the valuable evolution of real-world systems, from the continuously evolving patterns of social interactions and genetic pathways to the dynamic fluctuations of economic forces. Detecting communities for such evolving networks poses unique challenges. Detecting and analyzing the evolution of communities within dynamic graphs unlocks valuable insights into the underlying structural and temporal patterns of real-world systems. However, the sheer volume of modern graph data and the inherent complexity of the temporal dimension pose significant challenges to scalable community detection algorithms. Addressing this gap, our work explores the limited landscape of scalable distributed-memory parallel methods specifically designed for dynamic network community detection. We propose a novel parallel algorithm, DyG-DPCD (Dynamic Graph Distributed Parallel Community Detection), to detect communities in dynamic networks using the Message Passing Interface (MPI) framework. We present a vertex-centric approach, allowing us to detect communities through local optimization. Furthermore, we enhance our baseline algorithm by incorporating three heuristics, which improve the algorithm’s performance significantly while maintaining the quality of the solutions. We demonstrate the efficiency of our algorithm by experimenting on several real-world large-scale networks with hundreds of millions of edges spanning diverse domains. Notably, DyG-DPCD achieves speedups between 25× and 30× for large networks that we experimented on using NERSC compute nodes. In conclusion, our algorithm outperforms the STINGER parallel re-agglomeration algorithm by 30×.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Wave Tank Testing Report for Controls Validation of a Heaving Point Absorber

The core objectives of this project is to improve the power capture of three different wave energy conversion (WEC) devices by more than 50% using an advanced control system and validate the attained improvements using wave tank and full scale testing. In parallel, we will bring along the development of a wave prediction system that is required to enable effective control and test it at full scale. The purposes of this report are to: 1. Plan and document the 1/25th scale device testing at the wave-tank facility; 2. Document the test article, setup and methodology, sensor and instrumentation, mooring, electronics, wiring, and data flow and quality assurance; 3. Communicate the testing results between the associated members; 4. Facilitate reviews that will help to ensure all aspects (risk, safety, testing procedures, etc.); 5. Provide a systematic guide to setting up, executing and decommissioning the experiment.

16 TIDAL AND WAVE POWER↗
DOE OSTI2020

Turbulence-level dependence of cosmic ray parallel diffusion

ABSTRACT Understanding the transport of energetic cosmic rays belongs to the most challenging topics in astrophysics. Diffusion due to scattering by electromagnetic fluctuations is a key process in cosmic ray transport. The transition from a ballistic to a diffusive-propagation regime is presented in direct numerical calculations of diffusion coefficients for homogeneous magnetic field lines subject to turbulent perturbations. Simulation results are compared with theoretical derivations of the parallel diffusion coefficient’s dependences on the energy and the fluctuation amplitudes in the limit of weak turbulence. The present study shows that the widely used extrapolation of the energy scaling for the parallel diffusion coefficient to high turbulence levels predicted by quasi-linear theory does not provide a universally accurate description in the resonant-scattering regime. It is highlighted here that the numerically calculated diffusion coefficients can be polluted for low energies due to missing resonant interaction possibilities of the particles with the turbulence. Five reduced-rigidity regimes are established, which are separated by analytical boundaries derived in this work. Consequently, a proper description of cosmic ray propagation can only be achieved by using a turbulence-level-dependent diffusion coefficient and can contribute to solving the Galactic cosmic ray gradient problem.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Implementation of a Particle Accelerator Beam Dynamics Code on Multi-Node GPUs

Particle accelerators play an important role in a wide range of scientific discoveries and industrial applications. The self-consistent multi-particle simulation based on the particle-in-cell (PIC) method has been used to study charged particle beam dynamics inside those accelerators. However, the PIC simulation is time-consuming and needs to use modern parallel computers for high-resolution applications. In this paper, we implemented a parallel beam dynamics PIC code on multi-node hybrid architecture computers with multiple Graphics Processing Units (GPUs). We used two methods to parallelize the PIC code on multiple GPUs and observed that the replication method is a better choice for moderate problem size and current computer hardware while the domain decomposition method might be a better choice for large problem size and more advanced computer hardware that allows direct communications among multiple GPUs. Using the multi-node hybrid architectures at Oak Ridge Leadership Computing Facility (OLCF), the optimized GPU PIC code achieves a reasonable parallel performance and scales up to 64 GPUs with 16 million particles.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

Scaled ILU Smoothers for Navier-Stokes Pressure Projection

Incomplete LU (ILU) smoothers are effective in the algebraic multigrid (AMG) V-cycle for reducing high-frequency components of the error. However, the requisite direct triangular solves are comparatively slow on GPUs. Previous work has demonstrated the advantages of Jacobi iteration as an alternative to direct solution of these systems. Depending on the threshold and fill-level parameters chosen, the factors can be highly nonnormal and Jacobi is unlikely to converge in a low number of iterations. We demonstrate that row scaling can reduce the departure from normality, allowing us to replace the inherently sequential solve with a rapidly converging Richardson iteration. There are several advantages beyond the lower compute time. Scaling is performed locally for a diagonal block of the global matrix because it is applied directly to the factor. Further, an ILUT Schur complement smoother maintains a constant GMRES iteration count as the number of MPI ranks increases, and thus parallel strong-scaling is improved. Our algorithms have been incorporated into hypre, and we demonstrate improved time to solution for linear systems arising in the Nalu-Wind and PeleLM pressure solvers. For large problem sizes, GMRES+AMG executes at least five times faster when using iterative triangular solves compared with direct solves on massively parallel GPUs.

algebraic multigrid↗