Search NASA⌕ Search

DOE OSTI · 1016604

Moving Large Data Sets Over High-Performance Long Distance Networks

Abstract

In this project we look at the performance characteristics of three tools used to move large data sets over dedicated long distance networking infrastructure. Although performance studies of wide area networks have been a frequent topic of interest, performance analyses have tended to focus on network latency characteristics and peak throughput using network traffic generators. In this study we instead perform an end-to-end long distance networking analysis that includes reading large data sets from a source file system and committing large data sets to a destination file system. An evaluation of end-to-end data movement is also an evaluation of the system configurations employed and the tools used to move the data. For this paper, we have built several storage platforms and connected them with a high performance long distance network configuration. We use these systems to analyze the capabilities of three data movement tools: BBcp, GridFTP, and XDD. Our studies demonstrate that existing data movement tools do not provide efficient performance levels or exercise the storage devices in their highest performance modes. We describe the device information required to achieve high levels of I/O performance and discuss how this data is applicable in use cases beyond data movement performance.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Hodson, Stephen W, Poole, Stephen W, Ruwart, Thomas, Settlemyer, Bradley W. 2011-04-01. Moving Large Data Sets Over High-Performance Long Distance Networks. https://doi.org/10.2172/1016604. https://searchnasa.com/record/osti%3A1016604/moving-large-data-sets-over-high-performance-long-distance-networks

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related reports

Fire Protection Design

The purpose of this project is to design the demolition and installation of a deluge valve swap from a DV-5 to a DV-5A valve.

99 GENERAL AND MISCELLANEOUS↗