Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A Visual Analytics Framework for Reviewing Streaming Performance Data

Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.

Human-centered computing↗
DOE OSTI2022

Solving Unit Commitment Problems with Demand Responsive Loads: Preprint

This paper focuses on using variations of the Frank-Wolfe algorithm for solving unit commitment problems with high volumes of demand responsive loads on the power grid. We present a formulation of the unit commitment problem with demand responsive loads. We then show through reformulation and relaxations of the problem that variations of the Frank-Wolfe algorithm can be used to determine the time series decisions for the demand responsive loads. We show through computational experiments on the RTS-GMLC test system that the time series of demand responsive load decisions obtained through our approach are near optimal and provide details regarding how large scale parallel implementations of our approach can be highly computationally efficient.

demand response↗
DOE OSTI2022

Systems and methods for powder bed additive manufacturing anomaly detection

Detection and classification of anomalies for powder bed metal additive manufacturing. Anomalies, such as recoater blade impacts, binder deposition issues, spatter generation, and some porosities, are surface-visible at each layer of the building process. A multi-scaled parallel dynamic segmentation convolutional neural network architecture provides additive manufacturing machine and imaging system agnostic pixel-wise semantic segmentation of layer-wise powder bed image data. Learned knowledge is easily transferrable between different additive manufacturing machines. The anomaly detection can be conducted in real-time and provides accurate and generalizable results.

Scime, Luke R.↗
DOE OSTI2025

The Profiled Feldman-Cousins Method for Confidence Interval Construction for the Nova 3-Flavor Oscillation Analysis

The small interaction cross-section of neutrinos makes experimental neutrino physics particularly responsive to technological advancements. A significant development leveraged by the NOvA experiment is large-scale parallel processing, enabling novel computational approaches to longstanding experimental challenges. Central to managing the resulting high-throughput data is NOvA’s implementation of the Freight Train model, designed for efficient data production and handling.This dissertation details the methodology and execution of the NOvA 2024 3-Flavor Oscillation Analysis, supported by a comprehensive dataset spanning ten years. It emphasizes frequentist results refined through the Feldman-Cousins (FC) technique, specifically addressing confidence interval corrections in parameter estimation. The computational intensity associated with Feldman-Cousins arises from extensive Monte Carlo simulations, which were substantially mitigated through parallel computing on the Perlmutter supercomputer at the National Energy Research Scientific Computing Center (NERSC), employing the MPI framework.To further enhance computational efficiency, an Importance Sampling method is introduced and evaluated, demonstrating significant potential to reduce complexity, particularly in exploring extreme parameter space regions. This thesis presents both the successful application of advanced computational resources and the development of sophisticated statistical techniques, aiming to enhance the precision and scope of neutrino oscillation analyses.

Dye ajdye11190@gmail.com, Andrew Joseph [Mississip↗
DOE OSTI2024

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗
DOE OSTI2018

Efficient, Parallel At-scale Correlation Analysis for Atom Probe Tomography on Hybrid Architectures

Atom probe tomography (APT) is a material probing technique that has undergone dramatic improvements in its capability to map individual atoms within a material sample resulting in data files with hundreds of millions of atoms. Understanding the nano-structural features hidden in these massive amounts of atomic data is a crucial analysis task for materials scientists. However, fast analysis capabilities for large APT workloads remains a critical bottleneck. In this paper, we present the design, implementation and detailed performance evaluations of a parallel software capable of efficiently performing extremely time-consuming correlation analyses of massive high density APT data. Starting with shared memory implementations to motivate our design choices, we extend the implementation to hybrid architectures keeping realistic APT workloads in mind. Detailed performance analyses of three different parallel implementations of the software are supported by empirical results on a Cray XC30 and a Cray XC40 architecture. Its usefulness is demonstrated by reducing the turnaround time of an end-to-end APT correlation analysis on 100 millions atoms by three orders of magnitude using 2048 MPI ranks on 1024 nodes (24 cores per node) of a Cray XC30. The software reported here equips material scientists for the first time with a high-speed scalable capability for efficient and timely analyses of massive APT data.

Lu, Hao↗
NASA NTRS2002

Development and Applications of a Modular Parallel Process for Large Scale Fluid/Structures Problems

A modular process that can efficiently solve large scale multidisciplinary problems using massively parallel supercomputers is presented. The process integrates disciplines with diverse physical characteristics by retaining the efficiency of individual disciplines. Computational domain independence of individual disciplines is maintained using a meta programming approach. The process integrates disciplines without affecting the combined performance. Results are demonstrated for large scale aerospace problems on several supercomputers. The super scalability and portability of the approach is demonstrated on several parallel computers.

Guruswamy, Guru P.↗
NASA NTRS2001

Development and Applications of a Modular Parallel Process for Large Scale Fluid/Structures Problems

A modular process that can efficiently solve large scale multidisciplinary problems using massively parallel super computers is presented. The process integrates disciplines with diverse physical characteristics by retaining the efficiency of individual disciplines. Computational domain independence of individual disciplines is maintained using a meta programming approach. The process integrates disciplines without affecting the combined performance. Results are demonstrated for large scale aerospace problems on several supercomputers. The super scalability and portability of the approach is demonstrated on several parallel computers.

Guruswamy, Guru P.↗
DOE OSTI2020

Generating Massive Scale-free Networks: Novel Parallel Algorithms using the Preferential Attachment Model

Recently, there has been substantial interest in the study of various random networks as mathematical models of complex systems. As real-life complex systems grow larger, the ability to generate progressively large random networks becomes all the more important. This motivates the need for efficient parallel algorithms for generating such networks. Naïve parallelization of sequential algorithms for generating random networks is inefficient due to inherent dependencies among the edges and the possibility of creating duplicate (parallel) edges. In this article, we present message passing interface-based distributed memory parallel algorithms for generating random scale-free networks using the preferential-attachment model. Our algorithms are experimentally verified to scale very well to a large number of processing elements (PEs), providing near-linear speedups. The algorithms have been exercised with regard to scale and speed to generate scale-free networks with one trillion edges in 6 minutes using 1,000 PEs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Lab-Scale Cable-Driven Parallel Robot Prototype for Automated Prefabricated Component Manipulation

This paper presents the design and evaluation of a lab-scale cable-driven parallel robot (CDPR) developed as a flexible platform for automated installation of prefabricated components onto exterior building envelopes. Traditional manual installation methods for prefabricated components, which depend on scaffolding, cranes, cherry pickers, and verbal coordination, are not only labor-intensive and error-prone but also face significant limitations in dense urban environments due to site access constraints. To address these challenges, we developed a lab-scale CDPR platform capable of autonomously transporting building envelope components from a designated pickup zone to their target installation location, minimizing the need for human intervention. This study describes the system’s mechanical design, actuation architecture, real-time feedback system, and control strategy of the CDPR, and evaluates its performance in a laboratory environment. The robot’s actuation system uses torque control for end-effector manipulation. The robot’s real-time pose feedback comes from a construction-grade total station and a wireless inertial measurement unit (IMU), which together support precise end-effector control. Experimental results demonstrate the successful integration of the hardware, sensing, state estimation, and control subsystems. Preliminary tests showed that our lab-scale prototype can position the end effector with an error of less than 3 mm, which is a level of precision not previously achieved by existing CDPRs in construction applications. The key findings are twofold: (1) torque-only control is necessary but not sufficient for minimizing final pose error, and (2) incorporating real-time pose feedback can achieve the desired placement accuracy.

Liu, Yifang [Oak Ridge National Laboratory (ORNL),↗
DOE OSTI2023

LCIO

Performance of file systems shift during their life cycles. Evaluating this performance change over time is not trivial. Complexity arises in the interplay between external (i.e. application I/O workloads) and internal (i.e. the filesystem state) factors. Many benchmarks can test how a filesystem performs at the current snapshot state, but to observe the change over time necessitates that the filesystem state mutate (age) between benchmark runs. For a large-scale HPC parallel filesystem, the sheer scale and amount of interacting components during I/O operations magnify these challenges. LCIO addresses the question - how will the filesystem perform at different stages of its life cycle? LCIO is a synthetic benchmark, which provides the file system aging process to increase the amount of information that existing benchmarks like IOR and MDTest yield, as well as provide additional points of data that will be useful to system architects and engineers.

Bachstein, Matthew↗
NASA NTRS2000

HTMT-class Latency Tolerant Parallel Architecture for Petaflops Scale Computation

Computational Aero Sciences and other numeric intensive computation disciplines demand computing throughputs substantially greater than the Teraflops scale systems only now becoming available. The related fields of fluids, structures, thermal, combustion, and dynamic controls are among the interdisciplinary areas that in combination with sufficient resolution and advanced adaptive techniques may force performance requirements towards Petaflops. This will be especially true for compute intensive models such as Navier-Stokes are or when such system models are only part of a larger design optimization computation involving many design points. Yet recent experience with conventional MPP configurations comprising commodity processing and memory components has shown that larger scale frequently results in higher programming difficulty and lower system efficiency. While important advances in system software and algorithms techniques have had some impact on efficiency and programmability for certain classes of problems, in general it is unlikely that software alone will resolve the challenges to higher scalability. As in the past, future generations of high-end computers may require a combination of hardware architecture and system software advances to enable efficient operation at a Petaflops level. The NASA led HTMT project has engaged the talents of a broad interdisciplinary team to develop a new strategy in high-end system architecture to deliver petaflops scale computing in the 2004/5 timeframe. The Hybrid-Technology, MultiThreaded parallel computer architecture incorporates several advanced technologies in combination with an innovative dynamic adaptive scheduling mechanism to provide unprecedented performance and efficiency within practical constraints of cost, complexity, and power consumption. The emerging superconductor Rapid Single Flux Quantum electronics can operate at 100 GHz (the record is 770 GHz) and one percent of the power required by convention semiconductor logic. Wave Division Multiplexing optical communications can approach a peak per fiber bandwidth of 1 Tbps and the new Data Vortex network topology employing this technology can connect tens of thousands of ports providing a bi-section bandwidth on the order of a Petabyte per second with latencies well below 100 nanoseconds, even under heavy loads. Processor-in-Memory (PIM) technology combines logic and memory on the same chip exposing the internal bandwidth of the memory row buffers at low latency. And holographic storage photorefractive storage technologies provide high-density memory with access a thousand times faster than conventional disk technologies. Together these technologies enable a new class of shared memory system architecture with a peak performance in the range of a Petaflops but size and power requirements comparable to today's largest Teraflops scale systems. To achieve high-sustained performance, HTMT combines an advanced multithreading processor architecture with a memory-driven coarse-grained latency management strategy called "percolation", yielding high efficiency while reducing the much of the parallel programming burden. This paper will present the basic system architecture characteristics made possible through this series of advanced technologies and then give a detailed description of the new percolation approach to runtime latency management.

Sterling, Thomas↗
NASA NTRS2000

Applications of Parallel Process HiMAP for Large Scale Multidisciplinary Problems

HiMAP is a three level parallel middleware that can be interfaced to a large scale global design environment for code independent, multidisciplinary analysis using high fidelity equations. Aerospace technology needs are rapidly changing. Computational tools compatible with the requirements of national programs such as space transportation are needed. Conventional computation tools are inadequate for modern aerospace design needs. Advanced, modular computational tools are needed, such as those that incorporate the technology of massively parallel processors (MPP).

Guruswamy, Guru P.↗
DOE OSTI2017

Scaling Up Data-Parallel Analytics Platforms: Linear Algebraic Operation Cases

Linear algebraic operations such as matrix manipulations form the kernel of many machine learning and other crucial algorithms. Scaling up as well as scaling out such algorithms are key to supporting large scale data analysis that require efficient processing over millions of data samples. To this end, we present, ARION, a hardware acceleration based approach for scaling-up individual tasks of Spark, a popular data-parallel analytics platform. We support both linear algebraic operations of between two dense matrices, and between sparse and dense matrices in distributed environments. ARION provides a flexible control of acceleration according to matrix density, along with efficient scheduling based on runtime resource utilization. We demonstrate the benefit of our approach for general matrix multiplication operations over large matrices with up to four billion elements by using Gramian matrix computation that is commonly used in machine learning. Experiments show that our approach achieves more than 2× and 1.5× end-to-end performance speedups for dense and sparse matrices, respectively, and up to 57.04× faster computation compared to MLlib, a state of the art Spark-based implementation. This work is sponsored in part by the NSF under the grants: CNS-1565314, CNS-1405697, and CNS-1615411. The manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Xu, Luna↗
DOE OSTI2018

nbodykit: An Open-source, Massively Parallel Toolkit for Large-scale Structure

We present nbodykit, an open-source, massively parallel Python toolkit for analyzing large-scale structure (LSS) data. Using Python bindings of the Message Passing Interface, we provide parallel implementations of many commonly used algorithms in LSS. nbodykit is both an interactive and scalable piece of scientific software, performing well in a supercomputing environment while still taking advantage of the interactive tools provided by the Python ecosystem. Existing functionality includes estimators of the power spectrum, two- and three-point correlation functions, a friends-of-friends grouping algorithm, mock catalog creation via the halo occupation distribution technique, and approximate N-body simulations via the FastPM scheme. The package also provides a set of distributed data containers, insulated from the algorithms themselves, that enables nbodykit to provide a unified treatment of both simulation and observational data sets. nbodykit can be easily deployed in a high-performance computing environment, overcoming some of the traditional difficulties of using Python on supercomputers. We provide performance benchmarks illustrating the scalability of the software. The modular, component-based approach of nbodykit allows researchers to easily build complex applications using its tools. The package is extensively documented at http://nbodykit.readthedocs.io, which also includes an interactive set of example recipes for new users to explore. As open-source software, we hope nbodykit provides a common framework for the community to use and develop in confronting the analysis challenges of future LSS surveys.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Drosophila melanogaster pigmentation demonstrates adaptive phenotypic parallelism over multiple spatiotemporal scales

Abstract Populations are capable of responding to environmental change over ecological timescales via adaptive tracking. However, the translation from patterns of allele frequency change to rapid adaptation of complex traits remains unresolved. We used abdominal pigmentation in Drosophila melanogaster as a model phenotype to address the nature, genetic architecture, and repeatability of rapid adaptation in the field. We show that D. melanogaster pigmentation evolves as a highly parallel and deterministic response to shared environmental variation across latitude and season in natural North American populations. We then experimentally evolved replicate, genetically diverse fly populations in field mesocosms to remove any confounding effects of demography and/or cryptic structure that may drive patterns in wild populations; we show that pigmentation rapidly responds, in parallel, in fewer than 15 generations. Thus, pigmentation evolves concordantly in response to spatial and temporal climatic axes. We next examined whether phenotypic differentiation was associated with allele frequency change at loci with established links to genetic variance in pigmentation in natural populations. We found that across all spatial and temporal scales, phenotypic patterns were associated with variation at pigmentation-related loci, and the sets of genes we identified at each scale were largely nonoverlapping. Therefore, our findings suggest that parallel phenotypic evolution is associated with distinct components of the polygenic architecture shifting across each environmental axis to produce redundant adaptive patterns.

Evolutionary Biology↗
DOE OSTI2024

Bayesian Analysis of TRISO Fuel: Quantifying Model Inadequacy, Incorporating Lower-Length-Scale Effects, and Developing Parallel Active Learning Capabilities

The U.S. Department of Energy (DOE)’s Nuclear Energy Advanced Modeling and Simulation (NEAMS) program aims to develop predictive capabilities by applying computational methods to the analysis and design of advanced reactor and fuel-cycle systems. This program has been providing engineering-scale support for the continued development of BISON, a high-fidelity, high-resolution fuel performance tool. Fuel behavior in nuclear reactors is governed by a complex network of mechanisms that interact with various other physics aspects in the reactor system. Any model developed to represent fuel behavior will likely be idealized, resulting in uncertainties when comparing their predictions against the observed data. In Fiscal Year (FY)-23, we initiated the Uncertainty Quantification (UQ) work by using Bayesian methods to establish a level of model trustworthiness and further improve it, with a particular emphasis on TRI-Structural isOtropic (TRISO) nuclear fuel. This year, we further expanded on that UQ work by investigating an approach to quantifying model inadequacy and accounting for lower-length scale (LLS) effects in TRISO silver (Ag) release modeling. Furthermore, we are implementing parallel active learning capabilities to reduce the computational cost (i.e., required computational resources and elapsed time) of performing UQ. Specifically, we utilized The Kennedy O’Hagan framework for Bayesian uncertainty quantification (KOH) to account for model inadequacy in TRISO Ag release predictions made by BISON. The KOH framework represents an improvement over the standard Bayesian framework used in FY-23. Explicitly accounting for model inadequacy in the Bayesian framework helps establish the level of experimental noise uncertainty in the Advanced Gas Reactor (AGR) data. We compared the inverse UQ results obtained from both the standard Bayesian and KOH frameworks in light of the AGR-2/3/4 data, and also compared the predictive UQ results obtained from these two frameworks in light of the AGR-1 data. Next, we investigated the impact of considering LLS effects in the Ag release simulations. We developed an expanded database of LLS simulated effective diffusivities for Ag, covering a wide range of microstructures and temperatures. Using this database, we developed a framework for incorporating LLS effects into the engineering-scale Ag release UQ. We developed both parametric and non-parametric approaches for bridging the length scales. We then investigated the inverse UQ results in light of the AGR-2/3/4 data and the predictive UQ results in light of the AGR-1 data, and compared the LLS-informed approach and the Arrhenius equation, which does not include microstructure information. Finally, we discussed implementing parallel active learning capabilities in the Multiphysics Object Oriented Simulation Environment (MOOSE)/BISON to reduce the computational cost (i.e., computational resources and elapsed time) of Bayesian UQ. For verification purposes, we first tested these new capabil ities on a species interaction problem. We then demonstrated them on the TRISO Ag release application, showing that parallel active learning capabilities can enhance the accuracy of UQ while also substantially reducing the computational cost in comparison to the reference methods developed in FY-23.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗