Search NASA⌕ Search

SEARCH · Search NASA

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results reflect that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. Yet, the fault recovery time is 10% higher than the total data transfer time at any fault point.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Adaptive multi-level checkpointing

In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Magnetotelluric Data from Mountain Home, ID

This dataset includes magnetotelluric transfer functions in the form of EDI files for 16 stations collected by the USGS and 40 stations collected by Quantec Geoscience for Lawerence Berkeley National Lab around the Mountain Home area in Idaho. A 3D electrical resistivity model is included that images resistive and conductive bodies in the subsurface that maybe important for geothermal characterization. The model was created using ModEM using the high performance computer Yeti at the USGS.

15 GEOTHERMAL ENERGY↗
DOE OSTI2025

Data Management in the Continuum: Cross-facility Object-based Data Transfers

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

Bez, Jean Luca↗
DOE OSTI2019

FUSRAP Data Management and Information Transitions - 19459

The Formerly Utilized Sites Remedial Action Program (FUSRAP) was created in the mid-1970's to clean up radiological contamination resulting from early development of nuclear weapons. The US Department of Energy (US DOE) was responsible for FUSRAP until October 1997, when Congress transferred administration and execution of FUSRAP site cleanup to the US Army Corps of Engineers (USACE). Currently, the US DOE Office of Legacy Management (LM) determines the eligibility of a site for referral to USACE. USACE determines whether the site is included in FUSRAP for cleanup, conducts remedial actions, and transfers sites back to LM for long-term surveillance and maintenance (LTS and M). LM and USACE established the Data Management Working Group (DMWG) to (1) establish a common understanding of US DOE and USACE district-specific data, records practices, and requirements as they apply to FUSRAP programmatic documents for site transition and transfer; (2) establish methods to ensure accurate data transfer; and (3) develop a timeline for data and records transfer. These methods will ensure continuity of stewardship while transitioning from cleanup to LTS and M and beyond as more FUSRAP sites transfer from USACE to LM. The DMWG has met monthly and the meetings are attended by a wide range of LM and USACE project managers and subject matter experts. Discussion topics have addressed site records management, including data needs and uses; transfer and posting of Administrative Record and Permanent Record files for public accessibility; and management of legal and classified files. There have been a number of group successes, including the following: - Establishment of a common understanding of data and records practices and requirements as the sites transition and transfer from USACE to LM; - Development of effective lines of communication between USACE and LM concerning data management for transition; - Establishment of a cohesive approach to data management between the two agencies. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2025

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗
DOE OSTI2023

Wave Tank Characterization Data

This data set was collected from a 16k gallon single paddle type wave tank after adjustments were made to the transfer function in the control software. The file names describe the commanded wave height and period which can then be compared with the actual measured height and period of the generated waves.

16 TIDAL AND WAVE POWER↗
DOE OSTI2016

SIERRA Verification Module: Encore User Guide (V.4.40)

The Encore software package is both a stand-alone application and a software library. This guide explains the syntax of Encore input, provides examples, and is a comprehensive catalog of the Encore commands. Acting as a stand-alone application, Encore provides utilities for reading solutions from files and enables solution verification, postprocessing, field transfers, and basic mesh refinement. Acting as a software library, Encore is a component of the fluid, thermal, and solid modeling applications in the Sierra Mechanics suite. As a library, Encore provides the enclosing modeling application a superset of the stand-alone capabilities—enabled by application specific information—including physics specific postprocessors and adaptive mesh refinement.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Viability of S3 Object Storage for the ASC Program at Sandia

Recent efforts at Sandia such as DataSEA are creating search engines that enable analysts to query the institution’s massive archive of simulation and experiment data. The benefit of this work is that analysts will be able to retrieve all historical information about a system component that the institution has amassed over the years and make better-informed decisions in current work. As DataSEA gains momentum, it faces multiple technical challenges relating to capacity storage. From a raw capacity perspective, data producers will rapidly overwhelm the system with massive amounts of data. From an accessibility perspective, analysts will expect to be able to retrieve any portion of the bulk data, from any system on the enterprise network. Sandia’s Institutional Computing is mitigating storage problems at the enterprise level by procuring new capacity storage systems that can be accessed from anywhere on the enterprise network. These systems use the simple storage service, or S3, API for data transfers. While S3 uses objects instead of files, users can access it from their desktops or Sandia’s high-performance computing (HPC) platforms. S3 is particularly well suited for bulk storage in DataSEA, as datasets can be decomposed into object that can be referenced and retrieved individually, as needed by an analyst. In this report we describe our experiences working with S3 storage and provide information about how developers can leverage Sandia’s current systems. We present performance results from two sets of experiments. First, we measure S3 throughput when exchanging data between four different HPC platforms and two different enterprise S3 storage systems on the Sandia Restricted Network (SRN). Second, we measure the performance of S3 when communicating with a custom-built Ceph storage system that was constructed from HPC components. Overall, while S3 storage is significantly slower than traditional HPC storage, it provides significant accessibility benefits that will be valuable for archiving and exploiting historical data. There are multiple opportunities that arise from this work, including enhancing DataSEA to leverage S3 for bulk storage and adding native S3 support to Sandia’s IOSS library.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Introduction The agriculture genomics community has numerous data submission standards available, but the standards for describing and storing single-cell (SC, e.g., scRNA- seq) data are comparatively underdeveloped. Methods To bridge this gap, we leveraged recent advancements in human genomics infrastructure, such as the integration of the Human Cell Atlas Data Portal with Terra, a secure, scalable, open-source platform for biomedical researchers to access data, run analysis tools, and collaborate. In parallel, the Single Cell Expression Atlas at EMBL-EBI offers a comprehensive data ingestion portal for high-throughput sequencing datasets, including plants, protists, and animals (including humans). Developing data tools connecting these resources would offer significant advantages to the agricultural genomics community. The FAANG data portal at EMBL-EBI emphasizes delivering rich metadata and highly accurate and reliable annotation of farmed animals but is not computationally linked to either of these resources. Results Herein, we describe a pilot-scale project that determines whether the current FAANG metadata standards for livestock can be used to ingest scRNA-seq datasets into Terra in a manner consistent with HCA Data Portal standards. Importantly, rich scRNA-seq metadata can now be brokered through the FAANG data portal using a semi-automated process, thereby avoiding the need for substantial expert curation. We have further extended the functionality of this tool so that validated and ingested SC files within the HCA Data Portal are transferred to Terra for further analysis. In addition, we verified data ingestion into Terra, hosted on Azure, and demonstrated the use of a workflow to analyze the first ingested porcine scRNA-seq dataset. Additionally, we have also developed prototype tools to visualize the output of scRNA-seq analyses on genome browsers to compare gene expression patterns across tissues and cell populations. This JBrowse tool now features distinct tracks, showcasing PBMC scRNA-seq alongside two bulk RNA-seq experiments. Discussion We intend to further build upon these existing tools to construct a scientist-friendly data resource and analytical ecosystem based on Findable, Accessible, Interoperable, and Reusable (FAIR) SC principles to facilitate SC-level genomic analysis through data ingestion, storage, retrieval, re-use, visualization, and comparative annotation across agricultural species.

Genetics & Heredity↗
DOE OSTI2020

LLNL Kimberlina 1.2 NUFT Simulations June 2018 (v2)

This dataset contains the output 6,000, 3-dimensional reactive multi-phase flow and transport aquifer simulations of brine and CO2 leakage into a protective aquiver in California’s San Joaquin Valley and input data files detailing the geologic mesh, aquifer physical properties and CO2 and brine injection rates. This data set was generated as an ongoing effort with the US DOE National Risk Assessment Partnership (NRAP) to evaluate the effectiveness of monitoring techniques to detect brine and CO2 leakage from legacy wells into underground sources of drinking water overlaying a CO2 storage reservoir. Each simulation contains a unique set of input parameters, generated stochastically. The outputs consist of these upper three geologic layers (from top): the Etchegoin, Macoma-Chanac, Santa Margarita-McLure formations. These simulations span the several distances (1, 3 and 6 km or wells W31-0.2, W31-0.5 and W31-1.0, respectively) from the CO2 injector, initiated from bottom hole pressure and saturation to calculate wellbore leakage from the storage reservoir, with low and high regional groundwater gradients and wellbore leakage into 5 leaky nodes. The dataset includes 1,000 unique simulations for each distance, which each contain a unique aquifer heterogeneity, aquifer and caprock permeability, and two model generations are included with a high permeability (prod07) and hybrid permeability (prod09). The range of permeability distributions is listed in Table 1. Each model generation consists of 3,000 simulations. Included in the dataset are the leakage rates determined from 2D wellbore models which utilize the pressure and CO2 saturation from LBL's reservoir simulations, NUFT mesh files with distributed lithology, NUFT rocktab files which describe the material properties for the geologic layers and the NUFT input files and post-processed output 'ntab' files. Each ntab file contains spatial (rows) and temporal (columns) model output tables for each model cell, the locations (x,y,z) and dimensions for each cells (dx, dy, dz). Table 1. Permeability distribution ranges for prod07 and prod09 model generations Geologic Layer: Permeability Range (log10 m^2) prod07 prod09 Etchegoin -12.92 to -10.92 -13.70 to -11.44 Macoma-Chanac -12.72 to -10.72 -13.50 to -11.24 Santa Margarita-McLure -12.70 to -10.70 -13.48 to -11.22 The input files used to generate the model include which are included in the dataset are: Time series of CO2 leakage input into the model (ex: Q_brn.W31-0.2.sim1000.layers123.tab) Time series of CO2 leakage input into the model (ex: Q_CO2.W31-0.2.sim1000.layers123.tab) Physical properties of the aquifer materials detailing the aquifer porosity, solid density, partitioning coefficients, permeabilities and van-Genuchten parameters detailed in a NUFT rocktab file: (ex: sim1000.usnt.rocktab) Numerical mesh and geologic data assigned to each model cell detailed in a NUFT genmsh format (ex: sim1000.mesh_k16.prod07.trans.genmsh) The primary output parameters are: pH (use absolute value) Change in TDS (mg/kg) Change in Pressure (Pa) Change CO2 gas saturation (fraction range 0.0-1.0) for example, the directory /p/lscratchh/mansoor1/nrap/kimberlina/prod09/mainfiles/sim1000/W31- 0.2 contains: sim1000.W31-0.2.trans.pH.red.ntab sim1000.W31-0.2.no_bg.trans.TDS.red.ntab sim1000.W31-0.2.usnt.P.deltabg.red.ntab sim1000.W31-0.2.usnt.CO2_sat.deltabg.red.ntab Each row in the NTAB files consist of model output per numerical grid cell. Each output file contains 33 columns (variables), including the information of numerical records, geologic location and sizes and the simulated parameter values over time. The first 13 variables are about numerical records and relative geologic information for a simulation grid: 1. index: simulation index 2. i: the ith grid of x-axis 3. j: the ith grid of y-axis 4. k: the ith grid of z-axis 5. element_ref: element reference 6. nuft_ind: nuft index 7. x: grid location in the x axis direction 8. y: grid location in the y axis direction 9. z: grid location in the z axis direction 10. dx: grid length in the x axis direction 11. dy: grid length in the y axis direction 12. dz: grid length in the z axis direction 13. volume: volume of the simulation grid The remainder (14, 15, 16...) variables are the simulated parameter values over time, take Pressure as an example, are: 14. 0.0y: initial pressure per cell. 15. 10.0y: simulated pressure at the end of the 10th year. 16. 20.0y: simulated pressure at the end of the 20th year. ... (repeated for every 10 years until 200 years)... The model extends 10,000 m, 5,000 m and 1,411 m in the x,y and z dimensions, respectively. The mesh consists of 164,832 cells with mesh dimensions of 101 x 51 x 32 (nx, ny, nz), with cell dimensions ranging from 100 m laterally (along x and y-axis) and model layers are as designated in the z-axis: Layer 1: atmosphere (1e-30 m thick) Layer 2: upper caprock (10 m thick) Layers 3-13: Etchegoin (536.23 m thck) Layers 14-27: Macoma-Chanac (679.04 m thick) Layers 28-32: Santa Margarita-McLure (185.94 m thick) The wellbore is placed along node i=51, j=26, and extends vertically along 5 nodes from the top to the bottom of the model. Special instructions when extracting files: Each Gzip archive (ex: prod07.sim1000-sim00099.tar.gz) contains 100 simulations. Gzip archives should be transferred into base directories (ie. In Linux: mkdir prod07; mv prod07.*.tar.gz prod07/.) before extracting, or files will be overwritten. Each sub-simulation tree should have the following file structure pattern (using the linux 'tree' command): |-- prod07 | |-- sim0001 | |-- W31-0.2 | | |-- Q_brn.W31-0.2.sim0001.layers123.tab | | |-- Q_co2.W31-0.2.sim0001.layers123.tab | | |-- sim0001.W31-0.2.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-0.2.trans.pH.red.ntab | | |-- sim0001.W31-0.2.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-0.2.usnt.P.deltabg.red.ntab | |-- W31-0.5 | | |-- Q_brn.W31-0.5.sim0001.layers123.tab | | |-- Q_co2.W31-0.5.sim0001.layers123.tab | | |-- sim0001.W31-0.5.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-0.5.trans.pH.red.ntab | | |-- sim0001.W31-0.5.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-0.5.usnt.P.deltabg.red.ntab | |-- W31-1.0 | | |-- Q_brn.W31-1.0.sim0001.layers123.tab | | |-- Q_co2.W31-1.0.sim0001.layers123.tab | | |-- sim0001.W31-1.0.no_bg.trans.TDS.red.ntab | | |-- sim0001.W31-1.0.trans.pH.red.ntab | | |-- sim0001.W31-1.0.usnt.CO2_sat.deltabg.red.ntab | | |-- sim0001.W31-1.0.usnt.P.deltabg.red.ntab | |-- sim0001.mesh_k16.prod07.trans.genmsh Disclaimer This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes. Lawrence Livermore National Laboratory is operated by Lawrence Livermore National Security, LLC, for the U.S. Department of Energy, National Nuclear Security Administration under Contract DE-AC52-07NA27344. This report was reviewed and released as LLNL-MI-753464.

aquifer↗
DOE OSTI2019

Preparing the FUSRAP Colonie Interim Storage Site for Transfer Back to the US Department of Energy Office of Legacy Management - 19215

The Formerly Utilized Sites Remedial Action Program (FUSRAP) Colonie Interim Storage Site was transferred from the US Department of Energy (US DOE) to the US Army Corps of Engineers (USACE) in 1997. The USACE initiated remediation efforts at the Site under FUSRAP. The Site was separated into three distinct operable units (OU) to more efficiently manage the remedial process: 1) Groundwater OU, 2) Main Site Soils OU, and 3) Vicinity Property OU. The USACE has completed remedial actions at each of these three OUs and has advanced the OU remedies at the Site to the long-term stewardship stage to be executed by the US DOE Office of Legacy Management (LM). The US DOE LM is the current real property owner of record of the Colonie Main Site. A summary of each OU and the activities needed to prepare for site transfer follows: OU 1 - USACE issued the Colonie Groundwater OU Record of Decision (ROD) (USACE 2010a) on April 9, 2010. The Groundwater ROD remedy is monitored natural attenuation (MNA) and long-term monitoring (LTM) of groundwater after source removal completion. Results of groundwater monitoring indicate reductions in concentrations of contaminants of concern (COC). Based on the marked progress to date, USACE has optimized the groundwater LTM program (USACE, 2014) by reducing the well network, sampling frequency, and analytical parameters, and by abandoning wells that are no longer needed. A Groundwater ROD Five-Year Review report was submitted to the New York State Department of Environmental Conservation (NYSDEC) for review in 2018. NYSDEC agreed that MNA is still an appropriate remedial method although it is progressing at a slower rate than predicted. OU 2 - From 1999 to 2007, USACE completed the large-scale removal of soil at the Main Site, which included the adjacent Town of Colonie Vicinity Property. Radioactively contaminated soil along with accessible metals-contaminated soil and soil containing volatile organic compounds (VOC) were removed during this effort. A total of 103,402 cubic meters (m{sup 3}) of soil was excavated and disposed offsite. On March 26, 2015, USACE signed the Main Site Soils OU ROD (USACE, 2015a). The ROD remedy included provisions for Land Use Controls (LUC) consisting of institutional controls (IC) through an Environmental Easement for management of three small and discrete inaccessible locations within three of the 27 Final Status Survey units of the Main Site. The Environmental Easement provides a means of protection based on both current and future land use by placing restrictions on soil excavation and reuse. The easement and a supporting Site Management Plan (SMP) will be finalized by LM in coordination with NYSDEC. OU 3 - the Vicinity Property OU ROD was signed on 20 September 2017. This ROD declared No Action (NA) for dust and No Further Action (NFA) for soil and other media for the Vicinity Properties. Prior to the ROD, USACE performed dust sampling and other characterization activities at representative commercial and residential Vicinity Property locations. These efforts were reported in the Colonie FUSRAP Site Vicinity Property Operable Unit Remedial Investigation Summary Report (USACE, 2016). A Site Closeout Report (SCOR) was issued to NYSDEC and approved in 2018. The approval of this report starts the transfer process back to the LM. Ongoing and future transfer activities include records retention and management, preparation and filing of real estate records, physical site management (fencing, landscape maintenance, well abandonment) and coordination with state regulators and LM staff. This presentation is intended to make all stakeholders aware of the necessary steps for site transfer from the USACE back to the LM. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2022

NOvA Inclusive Nue CC Cross Section Data Release

NOvA inclusive electron neutrino cross section results presented at Neutrino2020Exposure:8.09E20 protons-on-target, neutrino-enhanced beamData release contains 3 ROOT files containing the double-differential measurement with respect to electron angle (cos theta) and electron energy, a single-differential measurement with respect to squared four-momentum transfer (Q2), and the total cross section as a function of neutrino energy. The 1D cross-section measurements include the same kinematic phase space restrictions present in the double-differential analysis.The files, xsec_{dThdE/Enu/dQ2}.root, contain the extracted cross section and associated covariance matrices for the double-differential cross section with respect to the observed electron kinematics, dThdE, neutrino energy, Enu, and squared four-momentum transfer, Q^2.In each file there will be the following histograms:— TH2D/TH1D xsec: unfolded measured cross section— TH2D xsec_cov: cross section covariance matrix— TH2D stat_cov: statistical covariance matrixIncludes text files, xsec_{dThdE/Enu/dQ2}.txt show the corresponding histograms in plain text format.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Distributed Computing for the Project 8 Experiment

The Project 8 collaboration aims to measure the absolute neutrino mass or improve on the current limit by measuring the tritium beta decay electron spectrum. We present the current distributed computing model for the Project 8 experiment. Project 8 is in its second phase of data taking with a near continuous data rate of 1Gbps. The current computing model uses DIRAC (Distributed Infrastructure with Remote Agent Control) for its workflow and data management. A detailed meta-data assignment using the DIRAC File Catalog is used to automate raw data transfers and subsequent stages of data processing. The DIRAC system is deployed on containers managed using a Kubernetes cluster to provide a scalable infrastructure. A modified DIRAC Site Director provides the ability to submit jobs using Singularity on opportunistic High-Performance Computing (HPC) sites.

Distributed Computing, Kubernetes, DIRAC, Project ↗
DOE OSTI2018

NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.

97 MATHEMATICS AND COMPUTING↗