Search NASA⌕ Search

SEARCH · Search NASA

Results for “data parallel”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,549 records · Page 86
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

Implementing Connected Component Labeling as a User Defined Operator for SciDB

We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDB's memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are "flattened" coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of O(1011) elements, with O(108) foreground cells and O(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.

UDO↗
DOE OSTI2023

Invited Paper: Benchmarking and Optimizing Data Movement on Emerging Heterogeneous Architectures

As supercomputers evolve, nodes are continually increasing in complexity. As a result, each generation of parallel systems brings new performance challenges. For instance, on recent systems inter-node communication has outperformed inter-socket, resulting in poor performance of many node-aware communication optimizations. Communication optimizations are critical for the performance and scalability of parallel applications, but are dependent on the parallel architecture, which varies significantly among recent generations of supercomputers. Furthermore, this paper investigates the performance of various paths of data movement on recent generations of systems, and analyzes the increased complexity of communication, particularly on recent heterogeneous systems. The paper also introduces MPI Advance, a communication library that enables optimizations to be created based on benchmark analysis of each emerging system.

benchmarking↗
DOE OSTI2022

A Hybrid Data-Driven and Model-Based Anomaly Detection Scheme for DER Operation: Preprint

This paper proposes a hybrid data and model-based anomaly detection for securing the operation of distributed energy resources (DERs) in distribution grids. Data-driven autoencoders (AE) are set up at the edge level by taking local DER data and detect anomalous operations by leveraging the reconstruction ability. In parallel, model-based state estimation (SE) is running at the system level by taking system models and measurements, the anomalies are identified by analyzing the measurements residual. The hybrid scheme preserves the benefits of both data-driven and model-based analysis and thus improves the robustness and accuracy of anomaly detection. It can be established by getting full use of the existing infrastructures in distribution grids. Numerical tests on a realistic distribution feeder in Southern California highlight the effectiveness as well as benefits of the proposed scheme.

anomaly detection↗
DOE OSTI2019

Counting Kmers for Biological Sequences at Large Scale

Counting the abundance of all the distinct kmers in biological sequence data is a fundamental step in bioinformatics. These applications include de novo genome assembly, error correction, etc. With the development of sequencing technology, the sequence data in a single project can reach Petabyte-scale or Terabyte-scale nucleotides. Counting demand for the abundance of these sequencing data is beyond the memory and computing capacity of single computing node, and how to process it efficiently is a challenge on a high-performance computing cluster. As such, we propose SWAPCounter, a highly scalable distributed approach for kmer counting. This approach is embedded with an MPI streaming I/O module for loading huge data set at high speed, and a counting bloom filter module for both memory and communication efficiency. By overlapping all the counting steps, SWAPCounter achieves high scalability with high parallel efficiency. In this work, the experimental results indicate that SWAPCounter has competitive performance with two other tools on shared memory environment, KMC 2 , and MSPKmerCounter. Moreover, SWAPCounter also shows the highest scalability under strong scaling experiments. In our experiment on Cetus supercomputer, SWAPCounter scales to 32,768 cores with 79% parallel efficiency (using 2048 cores as baseline) when processing 4 TB sequence data of 1000 Genomes. The source code of SWAPCounter is publicly available at https://github.com/mengjintao/SWAPCounter.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2016

An Adaptable Seismic Data Format

We present ASDF, the Adaptable Seismic Data Format, a modern and practical data format for all branches of seismology and beyond. The growing volume of freely available data coupled with ever expanding computational power opens avenues to tackle larger and more complex problems. Current bottlenecks include inefficient resource usage and insufficient data organization. Properly scaling a problem requires the resolution of both these challenges, and existing data formats are no longer up to the task. ASDF stores any number of synthetic, processed or unaltered waveforms in a single file. A key improvement compared to existing formats is the inclusion of comprehensive meta information, such as event or station information, in the same file. Additionally, it is also usable for any non-waveform data, for example, cross-correlations, adjoint sources or receiver functions. Last but not least, full provenance information can be stored alongside each item of data, thereby enhancing reproducibility and accountability. Any data set in our proposed format is self-describing and can be readily exchanged with others, facilitating collaboration. The utilization of the HDF5 container format grants efficient and parallel I/O operations, integrated compression algorithms and check sums to guard against data corruption. To not reinvent the wheel and to build upon past developments, we use existing standards like QuakeML, StationXML, W3C PROV and HDF5 wherever feasible. Usability and tool support are crucial for any new format to gain acceptance. We developed mature C/Fortran and Python based APIs coupling ASDF to the widely used SPECFEM3D_GLOBE and ObsPy toolkits.

58 GEOSCIENCES↗
DOE OSTI2025

Data for Kim et al., "Variations in the optical and molecular composition of dissolved organic matter exported from coastal wetlands"

Knowledge about sources and composition of marsh-derived dissolved organic matter (DOM) is critical for understanding the role of marshes in coastal biogeochemical cycling and the fate of marsh-derived DOM in the ocean. To investigate tidal variability in composition of marsh-derived DOM, Kim et al. examined the optical and molecular characteristics of hourly surface water samples at three tidal creeks in the Chesapeake Bay. Groundwater samples along the terrestrial landscape gradient as well as estuarine water from the adjacent estuary at each site were also collected to help resolve sources of surface water DOM. Samples were collected in summer 2024 at three sites – SWH: Sweet Hall Marsh, GCW: Kirkpatrick Marsh, and GWI: Goodwin Islands – which are part of synoptic sites in the Chesapeake Bay region of the COMPASS-FME (Coastal Observations, Mechanisms, and Predictions Across Systems and Scales - Field, Measurements, and Experiments) project. Surface water samples were collected hourly over a 48-hour period at each site. Groundwater and estuarine water samples were collected once. This dataset includes- Surface water depth and salinity- Dissolved organic carbon (DOC) and total dissolved nitrogen (TDN) concentrations- Optical indices and relative composition of parallel factor analysis (PARAFAC) components- High resolution mass spectrometry data.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1988

Modal identification using single-mode projection filters and comparison with ERA and MLE results

The Single-Mode Projection Filter (SPF) is a newly developed algorithm for eigensystem parameter identification from both analytical results and test data. The SPF is formulated with a single mode only and practical for parallel processing implementation. Explicit formulations of SPF are derived for the multi-input multi-output (MIMO) system by using the orthogonal matrices of the controllability and observability matrices in the general sense. The modal parameters of SPF are initially obtained from an analytical model in modal space. The experimental data are then processed through SPF to update its modal parameters and to minimize a cost function defined by the norm of an error matrix. The updated modal parameters represent the characteristics of the test data. A two-dimensional global minimum optimization algorithm is developed and applied for the filter update by using the interval analysis method. The SPF is developed based on a single-mode subsystem and identifies only one modal frequency and one modal damping within a specified region. For an n-modes structure, n SPF can be implemented for parallel processing to reduce the computational burden. The SPF is applied to analyze the simulated data for the MAST beam structure. The estimated modal parameters are comparable to those from the Eigensystem Realization Algorithm (ERA) and repeated modal frequencies are identified. The modal analysis of the Spacecraft Control Laboratory Experiment (SCOLE) data is also performed by using the ERA and the Maximum Likelihood Estimate (MLE). The result shows that the first five modal frequencies are very close from ERA and MLE. However, there are slight disparities in the damping rates and the computational burdens are quite different among these two algorithms.

Huang, Jen-Kuang↗
NASA NTRS1990

General-Purpose Data-Formatting Input/Output System

Multiplexable input/output (I/O) system developed as interface between host computer and real-time, fixed-base simulator cockpit. Unit designed to be general-purpose interface and operated through virtually any 8- or 16-bit, transistor/transistor-logic-level, parallel-I/O port with or without handshaking. Accepts byte-coded data in form of data blocks from host computer and routes data to various system I/O modules like discrete outputs, lamp drivers, and seven-segment-display drivers. Formats and routes data from input modules back to host computer. Other modules designed to drive alphanumeric displays and communications interfaces and to serve as analog-to-digital and digital-to-analog converter modules.

Busquets, Anthony M.↗
NASA NTRS1989

Efficient ICCG on a shared memory multiprocessor

Different approaches are discussed for exploiting parallelism in the ICCG (Incomplete Cholesky Conjugate Gradient) method for solving large sparse symmetric positive definite systems of equations on a shared memory parallel computer. Techniques for efficiently solving triangular systems and computing sparse matrix-vector products are explored. Three methods for scheduling the tasks in solving triangular systems are implemented on the Sequent Balance 21000. Sample problems that are representative of a large class of problems solved using iterative methods are used. We show that a static analysis to determine data dependences in the triangular solve can greatly improve its parallel efficiency. We also show that ignoring symmetry and storing the whole matrix can reduce solution time substantially.

Hammond, Steven W.↗
NASA NTRS1994

Low-freqency instabilities and the resulting velocity distributions of pickup ions at comet Halley

The interaction between the solar wind and newborn cometary ions is studied using a new analytical theory as well as one- and two-dimensional hybrid simulations. Using the observed parameters upstream of the comet Halley, a detailed study of wave excitation and the resulting particle distributions is presentd. Linear theory as well as simulations show that a variety of modes such as the fast magnetosonic mode, high frequency whistlers and obliquely propagating Alfven ion cyclotron waves can be excited. However, parallel propagating waves are found to be dominant in the wave spectrum and to control the scattering of the pickup ions. Several features of the observed distributions of pickup protons are explained. In particular, it is shown that the observed asymmetric pitch distribution for the pickup protons is due to the small saturation amplitude of the waves for the given parameters. Water group associated waves can lead to energy diffusion and further pitch angle scattering of protons. This effect is most likely to be important in the vicinity of the bow shock of comet Halley where the density of water group ions becomes comparable to that of protons. It is shown that the observed increase in the radius of the proton velocity shell just outside the bow shock can be due to water group waves. The nearby isotropic proton pitch angle distribution observed by Neugebauer et al. (1989) just outside the bow shock may, however, be related to the presence of a rotational discontinuity which has been identified in the magnetic field data. Just outside the bow shock, simulations show that parallel propagating water group waves can steepen with attached whistler wave packets. The steepening process at parallel propagation is a transient effect, in an important contrast to the case of steepening at oblique angles. The smaller beam densitites at comet Halley appears to be the main reason not only why waves at comet Halley have smaller amplitudes but also why oblique, steepening magnetosonic waves have not been detected at comet Halley, whereas they have been seen at comet Giacobini-Zinner. Halley, whereas they have been seen at comet Giacobini-Zinner.

Karimabadi, H.↗
DOE OSTI2017

A graph-based computational framework for simulation and optimisation of coupled infrastructure networks

Here, we present a computational framework that facilitates the construction, instantiation, and analysis of large-scale optimization and simulation applications of coupled energy networks. The framework integrates the optimization modeling package PLASMO and the simulation package DMNetwork (built around PETSc). These tools use a common graphbased abstraction that enables us to achieve compatibility between data structures and to build applications that use network models of different physical fidelity. We also describe how to embed these tools within complex computational workflows using SWIFT, which is a tool that facilitates parallel execution of multiple simulation runs and management of input and output data. We discuss how to use these capabilities to target coupled natural gas and electricity systems.

42 ENGINEERING↗
DOE OSTI2019

WISE-CASING: Frequency Domain EM Data from Cymric Oil Field, CA

Cymric oil field frequency domain electromagnetic (FEM) data with two parts of data labeled Part 1 and Part 2. Part 1 utilizes an electrical source line parallel to the receiver line, while Part 2 utilizes an electrical source line at a 75 deg angle to the receiver line. Data are plotted and compared to numerically modeled data to observe electric field response to well structure. In two configuration files are diagrams of the experiment layout.

15 GEOTHERMAL ENERGY↗
DOE OSTI2015

Delta: Data Reduction for Integrated Application Workflows

Integrated Application Workflows (IAWs) run multiple simulation workflow components concurrently on an HPC resource connecting these components using compute area resources and compensating for any performance or data processing rate mismatches. These IAWs require high frequency and high volume data transfers between compute nodes and staging area nodes during the lifetime of a large parallel computation. The available network band-width between the two areas may not be enough to efficiently support the data movement. As the processing power available to compute resources increases, the requirements for this data transfer will become more difficult to satisfy and perhaps will not be satisfiable at all since network capabilities are not expanding at a comparable rate. Furthermore, energy consumption in HPC environments is expected to grow by an order of magnitude as exascale systems become a reality. The energy cost of moving large amounts of data frequently will contribute to this issue. It is necessary to reduce the volume of data without reducing the quality of data when it is being processed and analyzed. Delta resolves the issue by addressing the lifetime data transfer operations. Delta removes subsequent identical copies of already transmitted data during transfers and restores those copies once the data has reached the destination. Delta is able to identify duplicated information and determine the most space efficient way to represent it. Initial tests show about 50% reduction in data movement while maintaining the same data quality and transmission frequency.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Airborne LiDAR to Improve Canopy Fuels Mapping for Wildfire Modeling

Increasing conflict between wildfire and the built environment has increased the need for more up-to-date and finer resolution canopy fuels data to improve wildfire modeling and associated risk forecasts. The US Forest Service and US Department of the Interior’s LANDFIRE product, which provides 30-m resolution canopy fuels data for the entire US, is one of the most widely used sources of fuels data. However, the last complete mapping effort for LANDFIRE is based on 2016 conditions, and subsequent updates reflect disturbances 1-2 years behind the release year. Airborne systems equipped with Light Detection and Ranging (LiDAR) sensors can be deployed to actively sense canopy structure and estimate canopy fuels data (cover, height, base height, bulk density) at finer resolutions. Canopy base height (CBH) and canopy bulk density (CBD) are difficult to measure both in the field and in LiDAR point clouds. Still, they are important for accurately modeling crown fires, which are often intense and difficult to contain. Additionally, point cloud datasets are large, and calculations require efficient utilization of computational resources. To address these challenges, we are working on an approach that uses openly available National Ecological Observatory Network (NEON) airborne LiDAR data, with calculations processed in the R programming language and parallelized through the lidR package. CBH and CBD are often derived from tree height, diameter at breast height, and species-specific allometries using the Fire and Fuels Extension of the Forest Vegetation Simulator (FFE-FVS). We aim to test if airborne LiDAR can estimate CBH and CBD without the use of empirical equations. Reliable estimates of canopy fuels data directly from airborne LiDAR could streamline quick, fine-resolution updates for use in wildfire behavior models.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2006

The Use of Dynamic Visual Acuity as a Functional Test of Gaze Stabilization Following Space Flight

After prolonged exposure to a given gravitational environment the transition to another is accompanied by adaptations in the sensorimotor subsystems, including the vestibular system. Variation in the adaptation time course of these subsystems, and the functional redundancies that exist between them make it difficult to accurately assess the functional capacity and physical limitations of astro/cosmonauts using tests on individual subsystems. While isolated tests of subsystem performance may be the only means to address where interventions are required, direct measures of performance may be more suitable for assessing the operational consequences of incomplete adaptation to changes in the gravitational environment. A test of dynamic visual acuity (DVA) is currently being used in the JSC Neurosciences Laboratory as part of a series of measures to assess the efficacy of a countermeasure to mitigate postflight locomotor dysfunction. In the current protocol, subjects visual acuity is determined using Landolt ring optotypes presented sequentially on a computer display. Visual acuity assessments are made both while standing and while walking at 1.8 m/s on a motorized treadmill. The use of a psychophysical threshold detection algorithm reduces the required number of optotype presentations and the results can be presented immediately after the test. The difference between the walking and standing acuity measures provides a metric of the change in the subject s ability to maintain gaze fixation on the visual target while walking. This functional consequence is observable regardless of the underlying subsystem most responsible for the change. Data from 15 cosmo/astronauts have been collected following long-duration (approx. 6 months) stays in space using a visual target viewing distance of 4.0 meters. An investigation of the group mean shows a change in DVA soon after the flight that asymptotes back to baseline approximately one week following their return to earth. The performance of some subjects nicely parallels the stereotypical recovery curve observed in the group mean data. Others show dramatic changes in DVA from one test day to another. These changes may be indicative of a re-adaptation process that is not characterized by a steady improvement with the passage of time, but is instead a dynamic search for appropriate coordinative strategy to achieve the desired gaze stabilization goal. Ground-based data have been collected in our lab using DVA with one of the goals being to improve the DVA test itself. In one of these studies, the DVA test was repeated using a visual target viewing distance of 0.5 meters. While walking, the relative contributions of the otoliths and semi-circular canals that are required to stabilize gaze are affected by visual target viewing distance. It may be possible to exploit this using the current treadmill DVA test to differentially assess changes in these vestibular subsystems. The postflight DVA evaluations currently used have been augmented to include the near target version of the test. Preliminary results from these assessments, as well as the results from the ground-based tests will also be reported. DVA provides a direct measure of a subject's ability to see clearly in the presence of self-motion. The use of the current tests for providing a functionally relevant metric is evident. However, it is possible to expand the scope of DVA testing to include scenarios other than walking. A facility for measuring DVA in the presence of passive movements is being created. Using a mechanized platform to provide the perturbation, it should be possible to simulate aircraft and automobile vibration profiles. Used in conjunction with the far and near visual displays this facility should be able to assess a subject s ability to clearly see distant objects as well as those that appear on the dashboard or instrument control panel during functionally relevant situations.

Peters, B. T.↗
NASA NTRS2020

Single-Photon Counting Detector Scalability for High Photon Efficiency Optical Communications Links

For high photon-efficiency deep space or low power optical communications links, such as the Orion Artemis-2 Optical Communications System (O2O) project, the received optical signal is attenuated to the extent that single- photon detectors are required. For direct-detection receivers operating at 1.55 µm wavelength, single-photon detectors including Geiger-mode InGaAs avalanche photon diodes (APDs), and in particular superconducting nanowire single-photon detectors (SNSPDs) offer the highest sensitivity and fastest detection speeds. However, these photon detectors exhibit a recovery time between registered input pulses, effectively reducing the detection efficiency over the recovery interval, resulting in missed photon detections, reduced count rate, and ultimately limiting the achievable data rate. A method to overcome this limitation is to divide the received optical signal into multiple detectors in parallel. Here we analyze this approach for a receiver designed to receive a high photon efficiency serially concatenated pulse position modulation (SCPPM) input waveform. From measured count rate and efficiency data using commercial SNSPDs, we apply a model from which we determine the effective detection efficiency, or blocking loss, for different input signal rates. We analyze the scalability of adding detectors in parallel for different modulation orders and background levels to achieve desired data rates. Finally we show tradeoffs between the number of detectors and the required received optical power, useful for real link design considerations.

Vyhnalek, Brian E.↗
DOE OSTI2019

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results reflect that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. Yet, the fault recovery time is 10% higher than the total data transfer time at any fault point.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1992

Modeling of supersonic combustor flows using parallel computing

While current 3D CFD codes and modeling techniques have been shown capable of furnishing engineering data for complex scramjet flowfields, the usefulness of such efforts is primarily limited by solutions' CPU time requirements, and secondarily by memory requirements. Attention is presently given to the use of parallel computing capabilities for engineering CFD tools for the analysis of supersonic reacting flows, and to an illustrative incompressible CFD problem using up to 16 iPSC/2 processors with single-domain decomposition.

Riggins, D.↗