DOE OSTI · 1433336
High Throughput WAN Data Transfer with Hadoop-Based Storage
Abstract
Hadoop distributed file system (HDFS) is becoming more popular in recent years as a key building block of integrated grid storage solution in the field of scientific computing. Wide Area Network (WAN) data transfer is one of the important data operations for large high energy physics experiments to manage, share and process datasets of PetaBytes scale in a highly distributed grid computing environment. In this paper, we present the experience of high throughput WAN data transfer with HDFS-based Storage Element. Two protocols, GridFTP and fast data transfer (FDT), are used to characterize the network performance of WAN data transfer.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Amin, A., Bockelman, B., Letts, J., Levshina, T., Martin, T., Pi, H., Sfiligoi, I., Thomas, M., Wurthwein, F.. 2011-01-01. High Throughput WAN Data Transfer with Hadoop-Based Storage. https://doi.org/10.1088/1742-6596%2F331%2F5%2F052016. https://searchnasa.com/record/osti%3A1433336/high-throughput-wan-data-transfer-with-hadoop-based-storage
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.