Search NASA⌕ Search

SEARCH · Search NASA

Results for “data parallel”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems

The growth in scientific data volumes has resulted in the need for new tools that enable users to operate on and analyze data on large-scale resources. In the last decade, a number of scientific workflow tools have emerged. These tools often target distributed environments, and often need expert help to compose and execute the workflows. Data-intensive workflows are often ad-hoc, they involve an iterative development process that includes users composing and testing their workflows on desktops, and scaling up to larger systems. In this paper, we present the design and implementation of Tigres, a workflow library that supports the iterative workflow development cycle of data-intensive workflows. Tigres provides an application programming interface to a set of programming templates i.e., sequence, parallel, split, merge, that can be used to compose and execute computational and data pipelines. We discuss the results of our evaluation of scientific and synthetic workflows showing Tigres performs with minimal template overheads (mean of 13 seconds over all experiments). We also discuss various factors (e.g., I/O performance, execution mechanisms) that affect the performance of scientific workflows on HPC systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2017

Triton: Igiugig Fish Video Analysis (Project Report)

This document summarizes the analysis of video data collected around the Ocean Renewable Power Company’s RivGen® device in the Kvichak River during July and August 2015. Human (manual) analysis of the data was undertaken and used to develop automated algorithms to detect fish in the frames and describe interaction behavior. In addition, a web application, EyeSea, was developed to combine manual and automated processing, so that ultimately the automated algorithms could be used to identify where human analysis was needed (i.e., when fish are present in video frames). The aim of the project was to develop algorithms that could identify video frames with fish present that could then be used to enable faster manual analysis. However, the project has also identified recommendations for future data collection, and research that could help analysis and interpretation. The manual analysis started to look at all data from the start of the deployment; this was to ensure rare events were seen, and initially focused on night-time data when more fish were present. This process highlighted the amount of time taken to identify fish, and ultimately only 42.33 hours of video were reviewed due to the time-consuming analysis. The data were classified as ‘Fish’, when the reviewer was confident it was a fish, and ‘Maybe’ fish, when it was difficult to distinguish. This was based on the movement, shape and color characteristics. ‘Fish’ were further classified as ‘adult’, ‘juvenile’, or ‘unidentifiable’. Behavioral attributes were noted, which were broadly divided into passive and avoidance activities. In over 42 hours data, there were only 20 potential contact interactions, of which 3 were ‘Maybe’ classifications, 12 were juveniles and 5 were adults; on only one occasion was an actual contact confirmed, and this was an adult fish with the camera, not the turbine itself. This also highlights the difficulties associated with confirming a strike or collision event as occurring or being a near-miss. More interactions were detected at night; this was probably biased by nighttime artificial light use, which may have attracted fish, but also could have increased detection probability as the light is reflected from the fish itself. For the algorithm development, background subtraction, optical flow, and deep learning techniques were considered. The deep learning approach was determined as needing too much training data for this application, and therefore was not continued. The optical flow analysis was considered promising, but did not give immediate results, so would need further investigation. Therefore, background subtraction was the main focus. Three methods of background subtraction were tried: Robust Principal Components Analysis (RPCA), Gaussian Mixture Model (GMM) and Video Background Extraction(ViBE). A classification technique was then applied to the foreground images to determine fish presence. Using this combination, it was found that fish could be accurately detected when occupying more pixels (>200 pixels, 98.2% correct; 100-200 pixels, 99.6% correct; 5-100 pixels, 85.4% correct; 2-5 pixels, 66.3% correct). In parallel, EyeSea was developed to convert the video data to a usable form and to enable manual and automated analysis of the data that would have a standardized output. Recommendations for further research, and standardized methods for data collection are given.

13 HYDRO ENERGY↗
DOE OSTI2022

SPACE: 3D parallel solvers for Vlasov-Maxwell and Vlasov-Poisson equations for relativistic plasmas with atomic transformations

A parallel, relativistic, three-dimensional particle-in-cell code SPACE has been developed for the simulation of electromagnetic fields, relativistic particle beams, and plasmas. In addition to the standard second-order Particle-in-Cell (PIC) algorithm, SPACE includes efficient novel algorithms to resolve atomic physics processes such as multi-level ionization of plasma atoms, recombination, and electron attachment to dopants in dense neutral gases. SPACE also contains a highly adaptive particle-based method, called Adaptive Particle-in-Cloud (AP-Cloud), for solving the Vlasov-Poisson problems. It eliminates the traditional Cartesian mesh of PIC and replaces it with an adaptive octree data structure. The code's algorithms, structure, capabilities, parallelization strategy, and performance have been discussed. Additionally, typical examples of SPACE applications to accelerator science and engineering problems are described.

43 PARTICLE ACCELERATORS↗
DOE OSTI2016

Interactive Correlation Analysis and Visualization of Climate Data

The relationship between our ability to analyze and extract insights from visualization of climate model output and the capability of the available resources to make those visualizations has reached a crisis point. The large volume of data currently produced by climate models is overwhelming the current, decades-old visualization workflow. The traditional methods for visualizing climate output also have not kept pace with changes in the types of grids used, the number of variables involved, and the number of different simulations performed with a climate model or the feature-richness of high-resolution simulations. This project has developed new and faster methods for visualization in order to get the most knowledge out of the new generation of high-resolution climate models. While traditional climate images will continue to be useful, there is need for new approaches to visualization and analysis of climate data if we are to gain all the insights available in ultra-large data sets produced by high-resolution model output and ensemble integrations of climate models such as those produced for the Coupled Model Intercomparison Project. Towards that end, we have developed new visualization techniques for performing correlation analysis. We have also introduced highly scalable, parallel rendering methods for visualizing large-scale 3D data. This project was done jointly with climate scientists and visualization researchers at Argonne National Laboratory and NCAR.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

With ever-increasing volumes of scientific data produced by high-performance computing applications, significantly reducing data size is critical because of limited capacity of storage space and potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are two leading BSD licensed open source C/C++ libraries for compressed floating-point arrays that support high throughput read and write random access. However, their performance is not consistent across different data sets and across different fields of some data sets, which raises the need for an automatic online (during compression) selection between SZ and ZFP, with minimal overhead. In this paper, the automatic selection optimizes the rate-distortion, an important statistical quality metric based on the signal-to-noise ratio. To optimize for rate-distortion, we investigate the principles of SZ and ZFP. We then propose an efficient online, low-overhead selection algorithm that predicts the compression quality accurately for two compressors in early processing stages and selects the best-fit compressor for each data field. Here, we implement the selection algorithm into an open-source library, and we evaluate the effectiveness of our proposed solution against plain SZ and ZFP in a parallel environment with 1,024 cores. Evaluation results on three data sets representing about 100 fields show that our selection algorithm improves the compression ratio up to 70 percent with the same level of data distortion because of very accurate selection (around 99 percent) of the bestfit compressor, with little overhead (less than 7 percent in the experiments).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

A Parallel Computing Framework for Finding the Supported Solutions to a Biobjective Network Optimization Problem

Abstract Solving multi‐objective combinatorial optimization problems can quickly become computationally challenging when applied to large networks generated from big data. Present‐day first‐rate Mixed Integer Programming (MIP) solvers have parallelism built‐in to take advantage of multicore architectures; but specialized network optimization algorithms that can often solve graph problems more efficiently than a general MIP solver are typically programmed serially. Thus, these network algorithm implementations do not take full advantage of modern multicore computer capabilities. Many parallel computing languages include a fork/join framework into their concurrency packages that allow for simple conversion of recursive serial algorithms to operate in parallel without having to use complicated low‐level message‐passing interfaces. Fork/join is particularly well suited to divide and conquer algorithms such as Non‐Inferior Set Estimation (NISE) method for computing the supported Pareto front of a multi‐objective optimization problem. This work develops a simple general parallel NISE (pNISE) implementation using the Java 7 concurrency tools, and then offers results from a specific implementation of solving a bi‐objective shortest path problem. The results indicate that this method is capable of sizeable speed‐ups on both small‐scale personal computers as well as large‐scale shared memory supercomputers. Finally, other network problems suitable to this method are discussed, including multi‐objective variants of the minimum spanning tree problem, transportation problem, assignment problem, maximum flow problem, and the minimum‐cost flow problem. Copyright © 2015 John Wiley & Sons, Ltd.

Medrano, Fernando Antonio↗
DOE OSTI2022

Elucidating the challenges in extracting ultra-slow flame speeds in a closed vessel - A CH 2 F 2 microgravity case study using optical and pressure-rise data

Refrigerants with a low global warming potential (GWP) possess mild flammability. Hence, a fundamental understanding of their combustion characteristics is required to assess their fire-hazardous potential. The laminar burning velocity is one fundamental property to describe these under safety evaluation aspects. However, measuring laminar burning velocities for slow-burning components, such as low-GWP alternatives, is challenging. This is due to influencing effects such as buoyant flame deformation, stretch, radiation, and confinement, especially at the flammability limits. Here, in the present study, we investigated near-limit flames of the representative refrigerant difluoromethane (CH 2 F 2 ) with nitrogen-enriched oxidizer mixtures to elucidate the potentials and limitations at ultra-slow flame speeds of two widely used flame speed measurement methods in the closed vessel configuration: the optical flame speed measurement in the early quasi-isobaric regime and the flame speed determination from the pressure-rise during the later phase of the isochoric combustion process. Experiments were performed under terrestrial- and microgravity, conducted in a specially designed setup suitable for drop tower applications, using both methods in parallel. Recommendations for the extraction of flame speed data are derived from the non-buoyant reference investigations under microgravity and transferred to ultra-slow flames at terrestrial gravity. Near-limit flames under microgravity were found to be significantly affected by stretch effects for the optical method so that flame speed extrapolations to unstretched conditions are prone to errors. Stretch also affects the pressure method’s lower evaluation limit, and errors can be estimated by combining flame speed data from the pressure method and Markstein lengths from optical experiments.

42 ENGINEERING↗
DOE OSTI2021

Athena: High-Performance Sparse Tensor Contraction Sequence on Heterogeneous Memory

Sparse tensor contraction sequence has been widely employed in many fields, such as chemistry and physics. However, how to efficiently implement the sequence faces multiple challenges, such as redundant computations and memory operations, massive memory consumption, and inefficient utilization of hardware. To address the above challenges, we introduce Athena, a high-performance framework for SpTC sequences. Athena introduces new data structures, leverages emerging Optane-based heterogeneous memory (HM) architecture, and stage parallelism. In particular, Athena introduces shared hash table-represented sparse accumulator to eliminate unnecessary input processing and data migration; Athena uses a novel data-semantic guided dynamic migration solution to make the best use of the Optane-based HM for high performance; Athena also co-runs execution phases with different characteristics to enable high hardware utilization. Evaluating with 12 datasets, we show that Athena brings 327-7362× speedup over the state-of-the-art SpTC algorithm. With the dynamic data placement guided by data semantics, Athena brings performance improvement on Optane-based HM over a state-of-the-art software-based data management solution, a hardware-based data management solution, and PMM-only by 1.58×, 1.82×, and 2.34× respectively.

Liu, Jiawen↗
DOE OSTI2020

Distributed-Memory Parallel Symmetric Nonnegative Matrix Factorization

We develop the first distributed-memory parallel implementation of Symmetric Nonnegative Matrix Factorization (SymNMF), a key data analytics kernel for clustering and dimensionality reduction. Our implementation includes two different algorithms for SymNMF, which give comparable results in terms of time and accuracy. The first algorithm is a parallelization of an existing sequential approach that uses solvers for non symmetric NMF. The second algorithm is a novel approach based on the Gauss-Newton method. It exploits second-order information without incurring large computational and memory costs. We evaluate the scalability of our algorithms on the Summit system at Oak Ridge National Laboratory, scaling up to 128 nodes (4,096 cores) with 70% efficiency. Additionally, we demonstrate our software on an image segmentation task.

Eswar, Srinivas↗
DOE OSTI2019

BCL: A Cross-Platform Distributed Data Structures Library

One-sided communication is a useful paradigm for irregular parallel applications, but most one-sided programming environments, including MPI's one-sided interface and PGAS programming languages, lack application-level libraries to support these applications. We present the Berkeley Container Library, a set of generic, cross-platform, high-performance data structures for irregular applications, including queues, hash tables, Bloom filters and more. BCL is written in C++ using an internal DSL called the BCL Core that provides one-sided communication primitives such as remote get and remote put operations. The BCL Core has backends for MPI, OpenSHMEM, GASNet-EX, and UPC++, allowing BCL data structures to be used natively in programs written using any of these programming environments. Along with our internal DSL, we present the BCL ObjectContainer abstraction, which allows BCL data structures to transparently serialize complex data types while maintaining efficiency for primitive types. We also introduce the set of BCL data structures and evaluate their performance across a number of high-performance computing systems, demonstrating that BCL programs are competitive with hand-optimized code, even while hiding many of the underlying details of message aggregation, serialization, and synchronization.

Brock, Benjamin↗
DOE OSTI2015

Parameter Estimation and Model Validation of Nonlinear Dynamical Networks

In the performance period of this work under a DOE contract, the co-PIs, Philip Gill and Henry Abarbanel, developed new methods for statistical data assimilation for problems of DOE interest, including geophysical and biological problems. This included numerical optimization algorithms for variational principles, new parallel processing Monte Carlo routines for performing the path integrals of statistical data assimilation. These results have been summarized in the monograph: “Predicting the Future: Completing Models of Observed Complex Systems” by Henry Abarbanel, published by Spring-Verlag in June 2013. Additional results and details have appeared in the peer reviewed literature.

42 ENGINEERING↗
DOE OSTI2024

Importance of Higher Fidelity Model Geometries during Optimization of Critical Experiments

PARADIGM, PARallel Approach of Differential and InteGral Measurements, is a cross-collaborative effort at Los Alamos National Laboratory between nuclear data theorists, differential and integral experimenters, as well as machine learning statisticians to tackle uncertainties in the intermediate region of 239 Pu. In essence, the idea behind PARADIGM is to remove the linear conceptualization of the nuclear data pipeline, shown in Figure 1, and replace it with a far more parallelized approach. The novel approach leverages machine learning to guide which differential measurements and integral experiments will result in the largest decrease in uncertain ties for a nuclide reaction pair in a given energy range. The concept builds off earlier work, EUCLID, which focused on the fast region of 239 Pu. The practical benefit of having evaluation, differential measurement, and integral experiment personnel in collaboration with machine learning is to represent the entire nuclear data in one snapshot. This enable large reduction in the time to deliver improved nuclear data, which using the PARADIGM approach could be done in 3 years. A general outline of PARADIGM and specific topics are available in other papers. The discussion here will pertain directly to the integral experiment design. More specifically, the process of taking a rough design and transforming it into a finalized neutronic model will be discussed.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Niche and neutral processes both shape community structure in parallelized, aerobic, single carbon-source enrichments

We present DNA sequence data in FASTA-formatted files from aerobic environmental microcosms inoculated with a sole carbon source. DNA sequences are of 16S rRNA genes present in DNA extracted from each microcosm along with the environmental samples (soil, water) used to inoculate them. These samples were sequenced using the Illumina MiSeq platform at the Environmental Sample Preparation and Sequencing Facility at Argonne National Laboratory. This data is compatible with standard microbiome analysis pipelines (e.g., QIIME, mothur, etc.).

Flynn, Theodore M.↗
DOE OSTI2016

Assessment of the MPACT Resonance Data Generation Procedure

Currently, heterogeneous models are being used to generate resonance self-shielded cross-section tables as a function of background cross sections for important nuclides such as 235 U and 238 U by performing the CENTRM (Continuous Energy Transport Model) slowing down calculation with the MOC (Method of Characteristics) spatial discretization and ESSM (Embedded Self-Shielding Method) calculations to obtain background cross sections. And then the resonance self-shielded cross section tables are converted into subgroup data which are to be used in estimating problem-dependent self-shielded cross sections in MPACT (Michigan Parallel Characteristics Transport Code). Although this procedure has been developed and thus resonance data have been generated and validated by benchmark calculations, assessment has never been performed to review if the resonance data are properly generated by the procedure and utilized in MPACT. This study focuses on assessing the procedure and a proper use in MPACT.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2025

Global tuning of hadronic interaction models with accelerator-based and astroparticle data

In high-energy and astroparticle physics, event generators play an essential role, even in the simplest data analyses. As analysis techniques become more sophisticated, e.g. based on deep neural networks, their correct description of the observed event characteristics becomes even more important. Physical processes occurring in hadronic collisions are simulated within a Monte Carlo framework. A major challenge is the modeling of hadron dynamics at low momentum transfer, which includes the initial and final phases of every hadronic collision. QCD-inspired phenomenological models used for these phases cannot guarantee completeness or correctness over the full phase space. These models usually include parameters which must be tuned to suitable experimental data. Until now, event generators have been developed and tuned mainly on the basis of data from high-energy physics experiments at accelerators. The wealth of data available from the latest generation of astroparticle experiments has not yet been fully exploited, and in many cases is not satisfactorily described. Both kinds of data sets are complementary as astroparticle experiments provide sensitivity especially to hadrons produced nearly parallel to the collision axis and cover center-of-mass energies up to several hundred TeV, well beyond those reached at colliders so far. In this report, we provide an overview of state-of-the-art event generators and their tuning, including the most relevant inputs from high-energy accelerator and astroparticle experiments. We present a road map that shows, for the first time, how the unified tuning of event generators with accelerator-based and astroparticle data can be performed.

Albrecht, J. [Ruhr U., Bochum, RAPP Ctr.; Ruhr U.,↗
DOE OSTI2023

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗