Search NASA⌕ Search

SEARCH · Search NASA

Results for “data parallel”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Hierarchical calibration and validation for modeling bench-scale solvent-based carbon capture. Part 1: Non-reactive physical mass transfer across the wetted wall column

A hierarchical model calibration and validation are proposed here for quantifying the confidence level of mass transfer prediction using a computational fluid dynamics (CFD) model, where the solvent-based carbon dioxide (CO 2 ) capture is simulated and the predicted results are compared to the corresponding bench-scale experimental data. Two unit problems with an increasing level of complexity are proposed to break down the complex physical/chemical processes of solvent-based CO 2 capture into relatively simpler problems to separate the effects of physical transport and chemical reaction. This paper focuses on the calibration and validation of the first unit problem, i.e., the CO 2 mass transfer across a falling ethanolamine (MEA) film in the absence of chemical reaction. This problem is investigated both experimentally and numerically using non-reactive nitrous oxide (N 2 O) as a surrogate for CO 2 . To capture motion of the gas-liquid interface, a volume of fluid method is employed with a one-fluid formulation to compute the mass transfer between the two phases. Parallel bench-scale experiments are designed and conducted to validate and calibrate the CFD models using a general Bayesian calibration approach. Two important transport parameters, Henry's constant and gas diffusivity, are calibrated to produce the posterior distributions, which will be used as input for the second unit problem to address the chemical absorption of CO 2 across the MEA falling film, where both mass transfer and chemical reaction are involved. Finally, mass transfer coefficients predicted by CFD are also compared with those predicted by the traditional/empirical correlations under both steady and wavy falling film conditions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

High-performance data management for whole slide image analysis in digital pathology

When dealing with giga-pixel digital pathology in whole-slide imaging, a notable proportion of data records holds relevance during each analysis operation. For instance, when deploying an image analysis algorithm on whole-slide images (WSI), the computational bottleneck often lies in the input-output (I/O) system. This is particularly notable as patch-level processing introduces a considerable I/O load onto the computer system. However, this data management process could be further paralleled, given the typical independence of patch-level image processes across different patches. This paper details our endeavors in tackling this data access challenge by implementing the Adaptable IO System version 2 (ADIOS2). Our focus has been constructing and releasing a digital pathology-centric pipeline using ADIOS2, which facilitates streamlined data management across WSIs. Additionally, we’ve developed strategies aimed at curtailing data retrieval times. The performance evaluation encompasses two key scenarios: (1) a pure CPU-based image analysis scenario (“CPU scenario”), and (2) a GPU-based deep learning framework scenario (“GPU scenario”). Our findings reveal noteworthy outcomes. Under the CPU scenario, ADIOS2 showcases an impressive two-fold speed-up compared to the brute-force approach. In the GPU scenario, its performance stands on par with the cutting-edge GPU I/O acceleration framework, NVIDIA Magnum IO GPU Direct Storage (GDS). From what we know, this appears to be among the initial instances, if any, of utilizing ADIOS2 within the field of digital pathology. The source code has been made publicly available at https://github.com/hrlblab/adios.

Wang, Xiao↗
DOE OSTI2018

Big Data Mining for Assessing Calibration of Building Energy Models

Residential and commercial buildings in China, India, the United States (US), United Kingdom (UK), and Italy consume 39-45% of each nation's primary energy. Building energy models can be used to automatically optimize the return-on-investment for retrofits to improve a building’s energy efficiency. However, with an average of 3,000 building descriptors necessary to accurately simulate a single building, there is a market need to reduce the transaction cost for creating a more robust model for simulating every building in a city and accurately estimate savings prior to capital expenditures.We used two of the world’s fastest supercomputers, assembled unique datasets, and developed innovative algorithms for big data mining to assess different methods to create an accurate building energy model. The team has leveraged a total of eight high performance computing resources and 218 servers to analyze the best methods, metrics, and algorithms for creating an accurate building energy model beyond current industry standards necessary for private-sector financing. The project developed the world’s fastest buildings simulator, has completed over 8 million simulations totaling over 300TB, and mined this data with over 130,000 parallel artificial intelligence algorithms. This was used to quantify accuracy of AI-developed calibration algorithms with results that surpass industry standard guidelines and can identify individual building parameters to between 15% and 32% of their actual value.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

The PARADIGM Project: Case Study in Balancing Experiment Uncertainty with Design simplicity

Accurate nuclear data are required for simulations of many applications including nuclear criticality safety. Actinide nuclear data at intermediate energies (from 1 to 100s of keV) are imprecise and inaccurate, because of scarce differential data, and an insufficient theory approach to capture the structures expected in the data to yield evaluated nuclear data, and lack of integral data for proper validation. This is a known deficiency but has proved challenging to address. More specifically, only 5% of integral experiments in the International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmark suite address intermediate energies (Fig. 1). Associated calculated effective multiplication factor, k eff , values for these experiments are far outside the experimental uncertainties and are 25× further from experiment than for fast energies. These differences could either stem from systematic biases in nuclear data, experiments or both. The goal of the PARADIGM (PARallel Approach of Differential and InteGral Measurements) project is to significantly reduce (by more than tens of percent) the uncertainties of intermediate energy actinide nuclear data. The PARADIGM project designed and intends to execute LANSCE (Los Alamos Neutron Science CEnter) and NCERC (National Criticality Experiments Research Center) intermediate experiments in parallel. They will specifically address a high priority nuclear data need—reducing bias and uncertainty in intermediate plutonium nuclear data. The two experiment will achieve that by informing each other and nuclear theory. By doing all these steps in parallel, the timeline to deliver improved nuclear data to users will significantly be reduced. This work will focus on the integral experiment final design and the balance of design and modeling simplicity while minimizing experiment uncertainty.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗
DOE OSTI2017

PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case

We describe an efficient parallel implementation of the selected inversion algorithm for distributed memory computer systems, which we call PSelInv. The PSelInv method computes selected elements of a general sparse matrix Athat can be decomposed as A= LU, where L is lower triangular and U is upper triangular. The implementation described in this article focuses on the case of sparse symmetric matrices. It contains an interface that is compatible with the distributed memory parallel sparse direct factorization SuperLU-DIST. Yet, the underlying data structure and design of PSelInv allows it to be easily combined with other factorization routines, such as PARDISO. We discuss general parallelization strategies such as data and task distribution schemes. Specifically, we describe how to exploit the concurrency exposed by the elimination tree associated with the LU factorization of A. We demonstrate the efficiency and accuracy of PSelInv by presenting several numerical experiments. In particular, we show that PSelInv can run efficiently on more than 4,000 cores for a modestly sized matrix. We also show how PSelInv can be used to accelerate large-scale electronic structure calculations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Dynamic Load Balancing Based on Constrained K-D Tree Decomposition for Parallel Particle Tracing

Particle tracing is a fundamental technique in flow field data visualization. In this work, we present a novel dynamic load balancing method for parallel particle tracing. Specifically, we employ a constrained k-d tree decomposition approach to dynamically redistribute tasks among processes. Each process is initially assigned a regularly partitioned block along with duplicated ghost layer under the memory limit. During particle tracing, the k-d tree decomposition is dynamically performed by constraining the cutting planes in the overlap range of duplicated data. This ensures that each process is reassigned particles as even as possible, and on the other hand the new assigned particles for a process always locate in its block. Result shows good load balance and high efficiency of our method.

Zhang, Jiang↗
DOE OSTI2020

A Visual Analytics Framework for Reviewing Streaming Performance Data

Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.

Human-centered computing↗
DOE OSTI2018

NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

This paper studies the hierarchical clustering problem, where the goal is to produce a dendrogram that represents clusters at varying scales of a data set. We propose the ParChain framework for designing parallel hierarchical agglomerative clustering (HAC) algorithms, and using the framework we obtain novel parallel algorithms for the complete linkage, average linkage, and Ward's linkage criteria. Compared to most previous parallel HAC algorithms, which require quadratic memory, our new algorithms require only linear memory, and are scalable to large data sets. ParChain is based on our parallelization of the nearest-neighbor chain algorithm, and enables multiple clusters to be merged on every round. We introduce two key optimizations that are critical for efficiency: a range query optimization that reduces the number of distance computations required when finding nearest neighbors of clusters, and a caching optimization that stores a subset of previously computed distances, which are likely to be reused. Experimentally, we show that our highly-optimized implementations using 48 cores with two-way hyper-threading achieve 5.8--110.1x speedup over state-of-the-art parallel HAC algorithms and achieve 13.75--54.23x self-relative speedup. Compared to state-of-the-art algorithms, our algorithms require up to 237.3x less space. Our algorithms are able to scale to data set sizes with tens of millions of points, which existing algorithms are not able to handle.

Computer Science↗
DOE OSTI2016

Thermally determining flow and/or heat load distribution in parallel paths

A method including obtaining calibration data for at least one sub-component in a heat transfer assembly, wherein the calibration data comprises at least one indication of coolant flow rate through the sub-component for a given surface temperature delta of the sub-component and a given heat load into said sub-component, determining a measured heat load into the sub-component, determining a measured surface temperature delta of the sub-component, and determining a coolant flow distribution in a first flow path comprising the sub-component from the calibration data according to the measured heat load and the measured surface temperature delta of the sub-component.

42 ENGINEERING↗
DOE OSTI2017

Systems and methods for rapid processing and storage of data

Systems and methods of building massively parallel computing systems using low power computing complexes in accordance with embodiments of the invention are disclosed. A massively parallel computing system in accordance with one embodiment of the invention includes at least one Solid State Blade configured to communicate via a high performance network fabric. In addition, each Solid State Blade includes a processor configured to communicate with a plurality of low power computing complexes interconnected by a router, and each low power computing complex includes at least one general processing core, an accelerator, an I/O interface, and cache memory and is configured to communicate with non-volatile solid state memory.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Thermally determining flow and/or heat load distribution in parallel paths

A method including obtaining calibration data for at least one sub-component in a heat transfer assembly, wherein the calibration data comprises at least one indication of coolant flow rate through the sub-component for a given surface temperature delta of the sub-component and a given heat load into said sub-component, determining a measured heat load into the sub-component, determining a measured surface temperature delta of the sub-component, and determining a coolant flow distribution in a first flow path comprising the sub-component from the calibration data according to the measured heat load and the measured surface temperature delta of the sub-component.

42 ENGINEERING↗
DOE OSTI2019

Systems and methods for rapid processing and storage of data

Systems and methods of building massively parallel computing systems using low power computing complexes in accordance with embodiments of the invention are disclosed. A massively parallel computing system in accordance with one embodiment of the invention includes at least one Solid State Blade configured to communicate via a high performance network fabric. In addition, each Solid State Blade includes a processor configured to communicate with a plurality of low power computing complexes interconnected by a router, and each low power computing complex includes at least one general processing core, an accelerator, an I/O interface, and cache memory and is configured to communicate with non-volatile solid state memory.

Stalzer, Mark A.↗
DOE OSTI2021

Systems and methods for rapid processing and storage of data

Systems and methods of building massively parallel computing systems using low power computing complexes in accordance with embodiments of the invention are disclosed. A massively parallel computing system in accordance with one embodiment of the invention includes at least one Solid State Blade configured to communicate via a high performance network fabric. In addition, each Solid State Blade includes a processor configured to communicate with a plurality of low power computing complexes interconnected by a router, and each low power computing complex includes at least one general processing core, an accelerator, an I/O interface, and cache memory and is configured to communicate with non-volatile solid state memory.

Stalzer, Mark A.↗
DOE OSTI2022

CEAZ: Accelerating Parallel I/O Via Hardware-Algorithm Co-Designed Adaptive Lossy Compression

As supercomputers continue to grow to exa-scale, the amount of data that needs to be saved or transmitted is exploding. To this end, many previous works have studied using error-bounded lossy compressors to reduce the data size and improve the I/O performance. However, little work has been done for effectively offloading lossy compression onto FPGA-based SmartNICs to reduce the compression overhead. In this paper, we propose a hardware-algorithm co-design of efficient and adaptive lossy compressor for scientific data on FPGAs (called CEAZ) to accelerate parallel I/O. Our contribution is fourfold: (1) We propose an efficient Huffman coding approach that can adaptively update Huffman codewords online based on codewords generated offline (from a variety of representative scientific datasets). (2) We derive a theoretical analysis to support a precise control of compression ratio under an error-bounded compression mode, enabling accurate offline Huffman codewords generation. This also help us create a fixed-ratio compression mode for consistent throughput. (3) We develop an efficient compression pipeline by adopting cuSZ’s dual-quantization algorithm to our hardware use case. (4) We evaluate CEAC on five real-world datasets with both a single FPGA board and 256 nodes from Bridges2 supercomputer. Experiments show that CEAZ outperforms the second-best FPGA-based lossy compressor by 2× of throughput and 9.6× of compression ratio. It also improves MPI_File_write and MPI_Gather throughputs by up to 32.7× and 31.4×, respectively.

Zhang, Chengming↗
DOE OSTI2018

User-Oriented Modeling Tools for Advanced Hybrid and Climate-Appropriate Rooftop Air Conditioners

Hybrid unitary air conditioning systems offer a pathway to substantially reduce energy use and peak electrical demand for cooling, heating, and ventilation in commercial buildings. Hybrid air conditioners incorporate multiple subsystems that are carefully orchestrated to provide climate- and application-specific efficiency advantages. There are a multitude of hybrid system architectures, but common subsystems include: heat recovery ventilation, indirect evaporative cooling, desiccant dehumidification, variable speed fans, modulating dampers, and multi-stage or variable-speed vapor compression cooling. Categorically, hybrid systems can operate in numerous discrete modes. For example: indirect evaporative cooling may operate for periods when the subsystem provides adequate sensible cooling, then vapor compression cooling will be included when more cooling or dehumidification is necessary. Laboratory assessments, field studies, and simulations have demonstrated that hybrid unitary air conditioners could reduce energy use for cooling and ventilation by 30-90% depending on climate and application. Heretofore, it has been challenging - if not impossible - for practitioners to model hybrid air conditioners as part of building energy simulations; and the limitation has severely obstructed broader adoption of technologies in this class. In this project, we developed a new feature for EnergyPlus that enables modeling hybrid unitary air conditioning equipment for building energy simulations. This is a significant advancement for both theory and practice, and confers public benefit by enabling practitioners to evaluate this compelling efficiency technology as a part of building energy simulations. The feature is a black-box model that requires extensive performance data for each hybrid unitary product. In parallel, we also developed new features for the Technology Performance Exchange to enable manufacturers to submit performance data in a standard format that can be used with the hybrid unitary model in EnergyPlus. Additionally, through this project we expanded university educational resources, and university- manufacturing industry collaborations in the field of energy efficiency technology. Over two years, we involved 20 undergraduate students in ambitious research projects focused on modeling complex multi-mode mechanical systems, supported three mechanical engineering bachelor theses, established undergraduate apprenticeships with multiple industry partners, and involved those partners in the process of design, validation, and debugging for the new EnergyPlus feature. The EnergyPlus feature is described and discussed in an academic article, as well as in an engineering reference, and input/output reference documentation for EnergyPlus. The Technology Performance Exchange features are live and publicly accessible, our manufacturer partners are primed to submit initial product information and performance data to the exchange, and the EnergyPlus feature is scheduled for public release in Spring 2018 as a part of EnergyPlus v8.9.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗