Search NASA⌕ Search

SEARCH · Search NASA

Results for “data transfer nodes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Automated, reliable, and efficient continental-scale replication of 7.3 petabytes of computational simulation data: A case study

We report on our experiences replicating 7.3 petabytes (PB) of Earth System Grid Federation (ESGF) computational simulation data from Lawrence Livermore National Laboratory (LLNL) in California to Argonne National Laboratory (ANL) in Illinois and Oak Ridge National Laboratory (ORNL) in Tennessee—a task motivated by a need for increased reliability, capacity, and performance. This task presented significant challenges: the need to move 29 million files twice under time pressure from aging storage hardware; a source file system bottleneck limiting throughput to 1.5 GB/s; frequent site maintenance windows; and the need for complete reliability at scale. We addressed these challenges using a simple replication tool that invoked Globus to transfer large bundles of files while tracking progress in a database, dynamically rerouting transfers to work around maintenance periods and file system limitations. Under the covers, Globus organized transfers to make efficient use of the high-speed Energy Sciences network (ESnet) and the data transfer nodes deployed at participating sites, and also addressed security, integrity checking, and recovery from a variety of transient failures. This success demonstrates the considerable benefits that can accrue from the adoption of performant data replication infrastructure. The replication tool is available at https://github.com/esgf2-us/data-replication-tools.

Globus↗

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗

Accelerating shared file checkpoint with local burst buffers

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Gooding, Thomas↗

Adaptive multi-level checkpointing

In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.

97 MATHEMATICS AND COMPUTING↗

HelioSwarm: Swarm Mission Design in High Altitude Orbit for Heliophysics

Resolving the complex three-dimensional turbulent structures that characterize the solar wind requires contemporaneous spatially and temporally distributed measurements. HelioSwarm is a mission concept that will deploy multiple, co-orbiting satellites to use the solar wind as a natural laboratory for understanding the fundamental, universal process of plasma turbulence. The HelioSwarm transfer trajectory and science orbit use a lunar gravity assist to deliver the ESPA-class nodes attached to a large data transfer hub to a P/2 lunar resonant orbit. Once deployed in the science orbit, the free-flying, propulsive nodes use simple Cartesian relative motion patterns to establish baseline separations both along and across the solar wind flow direction.

lunar resonant orbit↗

HelioSwarm: Swarm Mission Design in High Altitude Orbit for Heliophysics

Resolving the complex three-dimensional turbulent structures that characterize the solar wind requires contemporaneous spatially and temporally distributed measurements. HelioSwarm is a mission concept that will deploy multiple, co-orbiting satellites to use the solar wind as a natural laboratory for understanding the fundamental, universal process of plasma turbulence. The HelioSwarm transfer trajectory and science orbit use a lunar gravity assist to deliver the ESPA-class nodes attached to a large data transfer hub to a P/2 lunar resonant orbit. Once deployed in the science orbit, the free-flying, propulsive nodes use simple Cartesian relative motion patterns to establish baseline separations both along and across the solar wind flow direction.

lunar resonant orbit↗

Data transfer acceleration

Data transfer acceleration includes receiving, by a data transfer accelerator in a first node of a plurality of nodes, from a second node of the plurality of nodes, a request for data in a second state, wherein the second node stores an instance of the data in a first state; generating a message including one or more operations to transform the data from the first state to the second state; and sending the message to the second node in response to the request.

Gutierrez, Anthony↗

A Holistic Algorithmic Approach to Improving Accuracy, Robustness, and Computational Efficiency for Atmospheric Dynamics

Atmospheric weather and climate models must perform simulations very quickly to be useful. Therefore, modelers have traditionally focused on reducing computations as much as possible. However, in our new era of increasingly compute-capable hardware, data movement is now the prohibiting expense. This study examines the computational benefits of a new algorithmic approach to modeling atmospheric dynamics on scales relevant to weather and climate simulation. Rather than minimizing computations, this new approach considers the larger problem more holistically, including spatial accuracy, temporal accuracy, robustness (i.e., oscillations), on-node efficiency, and internode data transfers together at once. Numerical experiments demonstrate how computations can be strategically increased to simultaneously address each of these constraints while reducing data movement to adapt to modern accelerated hardware. The new algorithm can achieve at times up to 80% peak floating point throughput in single precision on the Nvidia Tesla V100 GPU, where the traditional approach is shown to only achieve single-digit floating point efficiency. Further, the new algorithm is twice as fast as a standard Runge--Kutta time integrator, and high-order accuracy with Weighted Essentially Non-Oscillatory (WENO) limiting came at less than 30% additional runtime cost on a GPU, thus increasing the accuracy per degree of freedom.

54 ENVIRONMENTAL SCIENCES↗

Multiplexing technique for computer communications via satellite channels

Multiplexing scheme combines technique of dynamic allocation with conventional time-division multiplexing. Scheme is designed to expedite short-duration interactive or priority traffic and to delay large data transfers; as result, each node has effective capacity of almost total channel capacity when other nodes have light traffic loads.

Binder, R.↗

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗

An MPA-IO interface to HPSS

This paper describes an implementation of the proposed MPI-IO (Message Passing Interface - Input/Output) standard for parallel I/O. Our system uses third-party transfer to move data over an external network between the processors where it is used and the I/O devices where it resides. Data travels directly from source to destination, without the need for shuffling it among processors or funneling it through a central node. Our distributed server model lets multiple compute nodes share the burden of coordinating data transfers. The system is built on the High Performance Storage System (HPSS), and a prototype version runs on a Meiko CS-2 parallel computer.

Jones, Terry↗

An MPI-IO Interface to HPSS

This paper describes an implementation of the proposed MPI-IO standard for parallel I/O. Our system uses third-party transfer to move data over an external network between the processors where it is used and the I/O devices where it resides. Data travels directly from source to destination, without the need for shuffling it among processors or funneling it through a central node. Our distributed server model lets multiple compute nodes share the burden of coordinating data transfers. The system is built on the High Performance Storage System (HPSS), and a prototype version runs on a Meiko CS-2 parallel computer.

Parallel Processing↗

Protocol Architecture Model Report

NASA's Glenn Research Center (GRC) defines and develops advanced technology for high priority national needs in communications technologies for application to aeronautics and space. GRC tasked Computer Networks and Software Inc. (CNS) to examine protocols and architectures for an In-Space Internet Node. CNS has developed a methodology for network reference models to support NASA's four mission areas: Earth Science, Space Science, Human Exploration and Development of Space (REDS), Aerospace Technology. This report applies the methodology to three space Internet-based communications scenarios for future missions. CNS has conceptualized, designed, and developed space Internet-based communications protocols and architectures for each of the independent scenarios. The scenarios are: Scenario 1: Unicast communications between a Low-Earth-Orbit (LEO) spacecraft inspace Internet node and a ground terminal Internet node via a Tracking and Data Rela Satellite (TDRS) transfer; Scenario 2: Unicast communications between a Low-Earth-Orbit (LEO) International Space Station and a ground terminal Internet node via a TDRS transfer; Scenario 3: Multicast Communications (or "Multicasting"), 1 Spacecraft to N Ground Receivers, N Ground Transmitters to 1 Ground Receiver via a Spacecraft.

Dhas, Chris↗

Trade-Off Analysis Report

NASAs Glenn Research Center (GRC) defines and develops advanced technology for high priority national needs in communications technologies for application to aeronautics and space. GRC tasked Computer Networks and Software Inc. (CNS) to examine protocols and architectures for an In-Space Internet Node. CNS has developed a methodology for network reference models to support NASAs four mission areas: Earth Science, Space Science, Human Exploration and Development of Space (REDS), Aerospace Technology. CNS previously developed a report which applied the methodology, to three space Internet-based communications scenarios for future missions. CNS conceptualized, designed, and developed space Internet-based communications protocols and architectures for each of the independent scenarios. GRC selected for further analysis the scenario that involved unicast communications between a Low-Earth-Orbit (LEO) International Space Station (ISS) and a ground terminal Internet node via a Tracking and Data Relay Satellite (TDRS) transfer. This report contains a tradeoff analysis on the selected scenario. The analysis examines the performance characteristics of the various protocols and architectures. The tradeoff analysis incorporates the results of a CNS developed analytical model that examined performance parameters.

Dhas, Chris↗

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗

Single and Multi-Node Modeling of Direct, Submerged, and Self-Pressurization of A Cryogenic Propellant Tank Using Nodal Tools

The pressurization of cryogenic propellant tanks will always be an important process so long as cryogenic liquids are being considered as fuel sources or used for other in-space applications. Pressure control of the tank ullage is necessary for managing propellant flowrates to an engine or a receiver tank, and modeling of the process is used to predict the pressurant requirements and the amount of propellant boiloff. Direct ullage pressurization is the more traditional approach to tank pressurization, as the physics are straight-forward, and ample test (flight) data have been collected and analyzed over the past several decades. Submerged injection pressurization is an alternate method for tank pressurization and has been shown to reduce pressurant requirements, subcool the propellant, and reduce the risk of ullage collapse. Additionally, the pressurant gas entering the ullage is usually much colder when using the submerged pressurization approach, resulting in reduced propellant boiloff. These benefits are at the expense of vaporizing a small percentage of the propellent. Both tank pressurization methods are viable options for current and future space missions, and it is important to have the capability of analyzing the tank ullage conditions for both approaches. Our previous work has demonstrated the development of a Generalized Fluid System Simulation Program (GFSSP) model, which contains a thermodynamic equilibrium heat and mass transfer subroutine capable of effectively analyzing both direct and submerged pressurization systems [1-2]. This subroutine has most recently been enhanced to include the non-equilibrium effect of pressurant dissolution into the propellant. To date the ullage has always been represented as a single node, and although the simulated single-node temperatures have good comparison with the volume-averaged ullage temperatures computed from test data, the physics of the thermal stratification in the ullage were never captured, and adjustment factors in the model were required. The purpose of this paper is to introduce the development of a multi-node ullage model using GFSSP and to discuss the improvements of the simulated ullage temperature distribution and its resulting effects on ullage heat transfer processes. Test data from the Cryogenic Propellant Storage and Transfer Engineering Developmental Unit (CPST EDU) was used for model validation. For additional comparison, a Thermal Desktop (TD) model was also developed to analyze the CPST EDU direct ullage pressurization tests using both a single node and multi-node approach. The model includes the direct pressurant line, vent line, fill/drain line, and a TD FloCAD Compartment. The TD FloCAD Compartment is employed to represent the liquid and ullage as single volumes inside the tank, to include a liquid/vapor interface, and to generate network level objects such as lumps (analogous to nodes in GFSSP), paths, and ties between the fluid and thermal elements. An established heat load on the model tank was leveraged from a pre-existing higher-fidelity model correlated to CPST EDU test data.

pressurization↗