Search NASA⌕ Search

SEARCH · Search NASA

Results for “data transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Data Transferability and Collection Consistency in Marine Renewable Energy

Concerns about the potential effects of tidal turbines and wave energy devices on the marine environment continue to slow siting and permitting/consenting of single devices and arrays worldwide. While research studies and early results from post-installation monitoring over the past decade have informed interactions between marine renewable energy (MRE) devices, marine animals, and habitats, regulators still demonstrate considerable reluctance to accelerate the permitting/consenting process for devices and arrays. Furthermore, the MRE industry is struggling with the high costs of baseline assessments and post-installation monitoring, as well as long timelines for obtaining permits, which leads to uncertainty and risk related to project financing. Regulators require assessment and monitoring information to allow them to carry out the necessary analyses to describe, permit/consent, and manage the environmental risks associated with new MRE technologies and new uses of ocean space. One way to reduce risks to the industry and the environment and to allow for acceleration of the permitting/consenting process could be to transfer learning, analyses, and data sets from one country to another, among projects, and across jurisdictional boundaries. In addition, data are collected around early-stage MRE devices using many different methods, instruments, and measurement scales. If similar parameters and accessible methods of data collection were used for baseline assessments and post-installation monitoring around all early-stage devices and MRE developments, the results would be more readily comparable. This comparability would lead to a decrease in scientific uncertainty and support a common understanding of the risk of MRE devices to the marine environment. This in turn would facilitate more efficient and shorter permitting/consenting processes, which would decrease the financial risk for MRE project development. As a means of addressing the concept of transferring data (information, learning, analyses, and data sets) among projects and collecting data consistently, Annex IV has developed a data transferability process that has been socialized with the MRE community, which includes regulators, industry, developers, consultants, and researchers. The data transferability process consists of five components: 1. A Data Transferability Framework brings together data sets in an organized fashion, compares the applicability of each data set for use on other projects, and guides the process of data transfer 2. A Data Collection Consistency Table provides preferred measurement methods or processes, reporting units, and the most common methods of analysis or interpretation and use of data 3. A Monitoring Data Sets Discoverability Matrix allows a practitioner to discover data sets based on the approach presented in the Framework 4. Best Management Practices (BMPs) include five BMPs related to data transferability and collection consistency 5. An Implementation Plan presents an approach for implementing and applying the data transferability process. This report documents the background and development of the data transferability process and associated components and summarizes the next steps needed to successfully implement and apply the data transferability process.

16 TIDAL AND WAVE POWER↗

High-performance End-to-End Integrity Verification on Big Data Transfer

The scale of scientific data generated by experimental facilities and simulations in high-performance computing facilities has been proliferating with the emergence of IoT-based big data. In many cases, this data must be transmitted rapidly and reliably to remote facilities for storage, analysis, or sharing, for the Internet of Things (IoT) applications. Simultaneously, IoT data can be verified using a checksum after the data has been written to the disk at the destination to ensure its integrity. However, this end-to-end integrity verification inevitably creates overheads (extra disk I/O and more computation). Thus, the overall data transfer time increases. In this article, we evaluate strategies to maximize the overlap between data transfer and checksum computation for astronomical observation data. Specifically, we examine file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We analyze these pipelining approaches in the context of GridFTP, a widely used protocol for scientific data transfers. Theoretical analysis and experiments are conducted to evaluate our methods. The results show that block-level pipelining is effective in maximizing the overlap mentioned above, and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.

data integrity↗

Minimally buffered data transfers between nodes in a data communications network

Methods, apparatus, and products for minimally buffered data transfers between nodes in a data communications network are disclosed that include: receiving, by a messaging module on an origin node, a storage identifier, a origin data type, and a target data type, the storage identifier specifying application storage containing data, the origin data type describing a data subset contained in the origin application storage, the target data type describing an arrangement of the data subset in application storage on a target node; creating, by the messaging module, origin metadata describing the origin data type; selecting, by the messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the data subset; and transmitting, by the messaging module to the target node, the selected data subset for storing in the target application storage in dependence upon the target data type without temporarily buffering the data subset.

97 MATHEMATICS AND COMPUTING↗

Cross-geography scientific data transferring trends and behavior

Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

Liu, Zhengchun↗

Fault-tolerant bandwidth reservation strategies for data transfers in high-performance networks

Many next-generation e-science applications need fast and reliable transfer of large volumes of data with guaranteed performance, which is typically enabled by the bandwidth reservation service in high-performance networks. One prominent issue in such network environments with large footprints is that node and link failures are inevitable, hence potentially degrading the quality of data transfer. We consider two generic types of bandwidth reservation requests (BRRs) concerning data transfer reliability: (i) to achieve the highest data transfer reliability under a given data transfer deadline, and (ii) to achieve the earliest data transfer completion time while satisfying a given data transfer reliability requirement. We propose two periodic bandwidth reservation algorithms with rigorous optimality proofs to optimize the scheduling of individual BRRs within BRR batches. The efficacy of the proposed algorithms is illustrated through extensive simulations in comparison with scheduling algorithms widely adopted in production networks in terms of various performance metrics.

97 MATHEMATICS AND COMPUTING↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Big data transfer in large-scale scientific and business applications is increasingly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) via advance bandwidth reservation. Provisioning agents need to carefully schedule data transfer requests, compute network paths, and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized, could be simply wasted due to the exclusive access during the approved time window, and cause extra overhead and complexity for resource management. This calls for accurate performance prediction to reserve bandwidths that match actual needs and avoid over-provisioning. We employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements collected in the past several years from data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated testbeds. We first analyze the performance patterns in response to a comprehensive list of parameters in end-host systems, network connections, and data transfer applications, which motivate the use of machine learning and also help us identify the effects of latent factors. We then propose threshold- and clustering-based methods to eliminate negative effects of latent factors in data preprocessing and build a robust performance predictor based on customized domain-oriented loss functions. The performance of the proposed methods is verified by extensive experiments using SVR and RFR as well as theoretical analysis of the general performance bound.

97 MATHEMATICS AND COMPUTING↗

Miniature Coil for Wireless Power and Data Transfer through Aluminum

This paper presents the design and development of miniature coils for wireless power and data transfer through metal. Our coil has a total size of 15 mm × 13 mm × 6 mm. Experimental results demonstrate that we can harvest 440 mW through a 1 mm-thick aluminum plate. Aluminum and stainless-steel barriers of different thicknesses were used to characterize coil performance. Using a pair of the designed coils, we have developed a through-metal communication system to successfully transfer data through a 1 mm-thick aluminum plate. A maximum data rate of 100 bps was achieved using only harvested power. To the best of our knowledge, this is the first report that demonstrates power and data transfer through aluminum using miniature coils.

42 ENGINEERING↗

Distributed data transfers in CMS

The multi-tiered computing infrastructure of the CMS experiment at the LHC depends on the reliable and fast transfer of data between the different CMS computing sites. Data have to be transferred from the Tier-0 to the Tier-l sites for archival in a timely manner to avoid overflowing disk buffers at CERN. Data have to be transferred in bursts to all Tier-2 level sites for analysis as well as synchronized between the different Tier-l sites. The data transfer system is the key ingredient which enables the optimal usage of all distributed resources. The operation of the transfer system consists of monitoring and debugging of transfer issues to guarantee a timely delivery of data to all corners of the CMS computing infrastructure. Further task of transfer operation is to guarantee the consistency of the data at all sites, both on disk and on tape. Procedures to verify the consistency and to debug and repair problems will be discussed.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

The Role of Data Transfer on the Selection of a Single vs Multiple Mesh Architecture for Tightly Coupled Multiphysics Applications

Data transfer from one distinct mesh to another may be necessary in any number of applications, including prolongation operations supporting multigrid solution methods, spatial adaptation, remeshing, and arbitrary Lagrangian-Eulerian (ALE) and multiphysics simulation. This data transfer process is also referred to as remapping, rezoning and interpolation. Intermesh data transfer has the potential to introduce error into a simulation; the magnitude and importance of which depends on the transfer scenario and the algorithm used to perform the transfer. For a transient analysis, data transfer may occur many times during a simulation, with possible error accumulation at each transfer. The present study develops selected scenarios that illustrate data transfer error and how it might impact an analysis. This study examines remapping error by using static analytical functions to compare various remapping schemes. It also investigates the significance and nature of data transfer error for a simple multiphysics system involving a transient coupled system of partial differential equations. It concludes that remapping error can be significant both for static functions and for coupled multiphysics systems. Aggregate error is shown to be a function of remapping scheme, mesh coarseness, nature of the remapped function and mesh disparity. In cases of extreme mesh disparity, this study shows that remapping can lead to excessive error and even to solution instability. Further, this work motivates that remapping error should be included in the estimation of numerical error, if data transfer is employed in a numerical simulation.

99 GENERAL AND MISCELLANEOUS↗

Fencing data transfers in a parallel active messaging interface of a parallel computer

Fencing data transfers in a parallel active messaging interface (`PAMI`) of a parallel computer, the PAMI including data communications endpoints, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes coupled for data communications through the PAMI and through data communications resources including a deterministic data communications network, including initiating execution through the PAMI of an ordered sequence of active SEND instructions for SEND data transfers between two endpoints, effecting deterministic SEND data transfers; and executing through the PAMI, with no FENCE accounting for SEND data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all SEND instructions initiated prior to execution of the FENCE instruction for SEND data transfers between the two endpoints.

97 MATHEMATICS AND COMPUTING↗

Fencing data transfers in a parallel active messaging interface of a parallel computer

Fencing data transfers in a parallel active messaging interface (`PAMI`) of a parallel computer, the PAMI including data communications endpoints, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes coupled for data communications through the PAMI and through data communications resources including a deterministic data communications network, including initiating execution through the PAMI of an ordered sequence of active SEND instructions for SEND data transfers between two endpoints, effecting deterministic SEND data transfers; and executing through the PAMI, with no FENCE accounting for SEND data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all SEND instructions initiated prior to execution of the FENCE instruction for SEND data transfers between the two endpoints.

97 MATHEMATICS AND COMPUTING↗

Dynamic Adaptation for High-Performance Data Transfers

Characteristics of the communication infrastructure determine which action should be taken when tuning data transfer operations in order to obtain high transfer rates. Local area networks and wide area networks have different characteristics, so they demonstrate diverse features in terms of congestion, failure rate, and latency. In most cases, congestion is not a concern in dedicated high bandwidth networks. However, the latency wall in data transfers over high bandwidth connections is still an issue. Enough data should be obtained from the applications and storage layers for high throughput performance. Data transfer optimization has been deeply studied in the literature. However, many of the solutions require kernel level changes that are not preferred by most domain scientists. In this study, we concentrate on application level auto-tuning methodologies that are applied in user-space for better transfer performance. Using multiple data transfer streams is a common technique applied in application layer to increase the network bandwidth utilization [2,5,10]. Instead of a single connection at a time, multiple streams are opened for a single data transfer service. Larger bandwidth in a network is gained with less packet loss rate; concurrent data transfer operations that are initiated at the same time better utilize the network and system resources.

97 MATHEMATICS AND COMPUTING↗

Fencing data transfers in a parallel active messaging interface of a parallel computer

Fencing data transfers in a parallel active messaging interface (`PAMI`) of a parallel computer, the PAMI including data communications endpoints, each endpoint including a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task; the compute nodes coupled for data communications through the PAMI and through data communications resources including at least one segment of shared random access memory; including initiating execution through the PAMI of an ordered sequence of active SEND instructions for SEND data transfers between two endpoints, effecting deterministic SEND data transfers through a segment of shared memory; and executing through the PAMI, with no FENCE accounting for SEND data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all SEND instructions initiated prior to execution of the FENCE instruction for SEND data transfers between the two endpoints.

97 MATHEMATICS AND COMPUTING↗

Fencing data transfers in a parallel active messaging interface of a parallel computer

Fencing data transfers in a parallel active messaging interface (`PAMI`) of a parallel computer, the PAMI including data communications endpoints, each endpoint including a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task; the compute nodes coupled for data communications through the PAMI and through data communications resources including at least one segment of shared random access memory; including initiating execution through the PAMI of an ordered sequence of active SEND instructions for SEND data transfers between two endpoints, effecting deterministic SEND data transfers through a segment of shared memory; and executing through the PAMI, with no FENCE accounting for SEND data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all SEND instructions initiated prior to execution of the FENCE instruction for SEND data transfers between the two endpoints.

97 MATHEMATICS AND COMPUTING↗

The Multicore-Aware Data Transfer Middleware Project

Existing data movement tools are still bound by major inefficiencies when running on multicore systems. To address these inefficiencies and limitations, DOE’s Advanced Scientific Computing Research (ASCR) office has funded Fermilab and Brookhaven National Laboratory to collaboratively work on the Multicore-Aware Data Transfer Middleware (MDTM) project. MDTM aims to accelerate data movement toolkits on multicore systems. A prototype version of MDTM is currently undergoing evaluation and enhancement.

Wu, Wenji↗