Search NASA⌕ Search

SEARCH · Search NASA

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

Variable Delay Testing Using ONE

This paper investigates the effect of long and changing propagation delays on the performance of TCP file transfers. Tests are performed with machines that emulate communication from a low/medium-earth satellite to Earth by way of a geosynchronous satellite. As a result of these tests, we find that TCP is fairly robust to varying delays given a high enough TCP timer granularity. However, performance degrades noticeably for larger file transfers when a finer timer granularity is used. Such results have also been observed in previous simulations by other researchers, and thus, this work serves as an extension of those results.

Ishac, Joseph↗
DOE OSTI2019

Fluorescent Amplification of Next Generation Sequencing Libraries Tools (FA-NGS tools) v1

This is a software utility for calculating pooling volumes and graphing quality control data of next generation sequencing libraries amplified with fluorescent intercalating agents and monitored by real-time quantitative PCR (qPCR). The tool is used to calculate pooling volumes of individual next generation sequencing libraries based on experimentally determined fluorescence values. To do this, fluorescent values are used as a proxy for relative concentration to determine the ratio of transfer volumes. The output is a "transfer file" that specifies the transfer volume and destination well of a given library. The tool can also be used to assess the quality of individual next generation sequencing libraries by melting curve analysis. The output are small-multiples graphs of the melting curves of individual sequencing libraries according to the specified plate layout.

Garber, Megan↗
NASA NTRS1999

Acceleration Measurement and Characterization in Support of the USMP-4 Payloads

One common characteristic of the USMP-4 experiments is that various effects of gravity make it difficult, if not impossible, to achieve usable results when performing the experiments on Earth's surface. Therefore, the investigators took advantage of the microgravity environment afforded by being in low-Earth orbit to perform their research. Interpretation of the experiment results both during the mission and upon post-mission analyses of data and samples required an understanding of the microgravity environment in which the experiments were conducted. To achieve that understanding, data were collected using the Orbital Acceleration Research Experiment (OARE) and two Space Acceleration Measurement Systems (SAMS). Data from those systems, combined with an assessment of mission and experiment activities, were used to characterize the microgravity environment that existed on Columbia during the mission. The text herein gives details about some characteristics of the environment that were noted during the mission and during post-mission data analysis. The disturbances studied include the Ku-band antenna 17 Hz dither; the effect of changing the Orbiter attitude deadband limits; the effects of different bicycle ergometer configurations; and the effect of IDGE (Isothermal Dendritic Growth Experiment) experiment fans and SAMS computer hard drives. Additional information about the microgravity environment is provided. Supplementary data plots representing the environment throughout the majority of the mission are available at the Uniform Resource Locator (URL). Data files for both SAMS and OARE are accessible via anonymous file transfer protocol from the file server.

Rogers, M. J. B.↗
NASA NTRS1998

On-Board File Management and Its Application in Flight Operations

In this paper, the author presents the minimum functions required for an on-board file management system. We explore file manipulation processes and demonstrate how the file transfer along with the file management system will be utilized to support flight operations and data delivery.

on-board file management flight operations flight ↗
DOE OSTI2024

Complete and Correct Transfer of Information (CACTI)

Many distributed systems, file transfer mechanisms, and message passing systems offer reliability mechanisms such as acknowledgements, retries, and durability. While these tools may be “good enough” for their typical use cases, they may not offer sufficient coverage for the wide range of faults that impact data transfers and communication. A gap in the reliability measures may lead to some small amount of data loss. Some high-consequence systems cannot tolerate the loss or corruption of even a single record. We present seven principles that will counter a wide range of faults and protect against data loss and corruption. These principles bring together lessons learned from a wide range of technologies and can inform appropriate system design and application usage. These principles will help readers reason on how prevent data loss in a multi-hop pipeline and how to properly use tools that may have a deficiency in reliability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
NASA NTRS2001

Registered File Support for Critical Operations Files at (Space Infrared Telescope Facility) SIRTF

The SIRTF Science Center's (SSC) Science Operations System (SOS) has to contend with nearly one hundred critical operations files via comprehensive file management services. The management is accomplished via the registered file system (otherwise known as TFS) which manages these files in a registered file repository composed of a virtual file system accessible via a TFS server and a file registration database. The TFS server provides controlled, reliable, and secure file transfer and storage by registering all file transactions and meta-data in the file registration database. An API is provided for application programs to communicate with TFS servers and the repository. A command line client implementing this API has been developed as a client tool. This paper describes the architecture, current implementation, but more importantly, the evolution of these services based on evolving community use cases and emerging information system technology.

file management services↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed a couple of FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. During this talk, we are going to present the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment and giving an overview of the possible issues encountered. In addition, we discuss our method of certificate management and maintenance utilizing Kubernetes cronjobs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2002

Comparison of VDL Modes in the Aeronautical Telecommunications Network

VHF Digital Link (VDL) has been identified as a method of communication between aircraft and ground stations in the Aeronautical Telecommunications Network (ATN). Three different modes of VDL have been suggested for implementation. Simulations were conducted to compare the data transfer capabilities of VDL Modes 2, 3, and 4. These simulations focus on up to 50 aircraft communicating with a single VDL ground station. The data traffic is generated by the standard File Transfer Protocol (FTP) and Hyper Text Transfer Protocol (HTTP) applications in the aircraft. Comparisons of the modes are based on the number of files and pages transferred and the response time.

Bretmersky, Steven↗
NASA NTRS2001

The Standard Autonomous File Server, a Customized, Off-the-Shelf Success Story

The Standard Autonomous File Server (SAFS), which includes both off-the-shelf hardware and software, uses an improved automated file transfer process to provide a quicker, more reliable, prioritized file distribution for customers of near real-time data without interfering with the assets involved in the acquisition and processing of the data. It operates as a stand-alone solution, monitoring itself, and providing an automated fail-over process to enhance reliability. This paper will describe the unique problems and lessons learned both during the COTS selection and integration into SAFS, and the system's first year of operation in support of NASA's satellite ground network. COTS was the key factor in allowing the two-person development team to deploy systems in less than a year, meeting the required launch schedule. The SAFS system his been so successful, it is becoming a NASA standard resource, leading to its nomination for NASA's Software or the Year Award in 1999.

Semancik, Susan K.↗
NASA NTRS2015

Global Precipitation Measurement (GPM) Mission: Precipitation Processing System (PPS) GPM Mission Gridded Text Products Provide Surface Precipitation Retrievals

In February 2015, the Global Precipitation Measurement (GPM) mission core satellite will complete its first year in space. The core satellite carries a conically scanning microwave imager called the GPM Microwave Imager (GMI), which also has 166 GHz and 183 GHz frequency channels. The GPM core satellite also carries a dual frequency radar (DPR) which operates at Ku frequency, similar to the Tropical Rainfall Measuring Mission (TRMM) Precipitation Radar, and a new Ka frequency. The precipitation processing system (PPS) is producing swath-based instantaneous precipitation retrievals from GMI, both radars including a dual-frequency product, and a combined GMIDPR precipitation retrieval. These level 2 products are written in the HDF5 format and have many additional parameters beyond surface precipitation that are organized into appropriate groups. While these retrieval algorithms were developed prior to launch and are not optimal, these algorithms are producing very creditable retrievals. It is appropriate for a wide group of users to have access to the GPM retrievals. However, for researchers requiring only surface precipitation, these L2 swath products can appear to be very intimidating and they certainly do contain many more variables than the average researcher needs. Some researchers desire only surface retrievals stored in a simple easily accessible format. In response, PPS has begun to produce gridded text based products that contain just the most widely used variables for each instrument (surface rainfall rate, fraction liquid, fraction convective) in a single line for each grid box that contains one or more observations.This paper will describe the gridded data products that are being produced and provide an overview of their content. Currently two types of gridded products are being produced: (1) surface precipitation retrievals from the core satellite instruments GMI, DPR, and combined GMIDPR (2) surface precipitation retrievals for the partner constellation satellites. Both of these gridded products are generated for a.25 degree x.25 degree hourly grid, which are packaged into daily ASCII (American Standard Code for Information Interchange) files that can downloaded from the PPS FTP (File Transfer Protocol) site. To reduce the download size, the files are compressed using the gzip utility.This paper will focus on presenting high-level details about the gridded text product being generated from the instruments on the GPM core satellite. But summary information will also be presented about the partner radiometer gridded product. All retrievals for the partner radiometer are done using the GPROF2014 algorithmusing as input the PPS generated inter-calibrated 1C product for the radiometer.

precipitation↗
DOE OSTI2021

ESnet Secure Copy (EScp) v0.6

EScp is a high speed transfer tool with a similar command line syntax to scp. Unlike SCP it is designed to transfer files at high speed, thus far we have been able to show 100gbit/s transfers, although I expect that the throughput should scale in proportion to the network interface, i.e. I expect 400gbit/s performance on our 400gbit/s test bed. EScp achieves good performance through an innovative design (multithreaded, zero copy transfers), along with pluggable filters and I/O engines. As an example, you can switch from POSIX i/O to UIO by checking a different engine. It also natively supports encryption, and cheksums for file verification and transport security. AAA is through standard SSH (same as SCP). By taking advantage of filters, EScp supports transferring unstructured data and/or I/O to non-posix data sources. Examples include streaming data (i.e. from equipment), transferring data to the cloud, and/or supporting non-posix file systems (like HPSS).

Shiflett, Charles↗
NASA NTRS1996

Experiences From NASA/Langley's DMSS Project

There is a trend in institutions with high performance computing and data management requirements to explore mass storage systems with peripherals directly attached to a high speed network. The Distributed Mass Storage System (DMSS) Project at the NASA Langley Research Center (LaRC) has placed such a system into production use. This paper will present the experiences, both good and bad, we have had with this system since putting it into production usage. The system is comprised of: 1) National Storage Laboratory (NSL)/UniTree 2.1, 2) IBM 9570 HIPPI attached disk arrays (both RAID 3 and RAID 5), 3) IBM RS6000 server, 4) HIPPI/IPI3 third party transfers between the disk array systems and the supercomputer clients, a CRAY Y-MP and a CRAY 2, 5) a "warm spare" file server, 6) transition software to convert from CRAY's Data Migration Facility (DMF) based system to DMSS, 7) an NSC PS32 HIPPI switch, and 8) a STK 4490 robotic library accessed from the IBM RS6000 block mux interface. This paper will cover: the performance of the DMSS in the following areas: file transfer rates, migration and recall, and file manipulation (listing, deleting, etc.); the appropriateness of a workstation class of file server for NSL/UniTree with LaRC's present storage requirements in mind the role of the third party transfers between the supercomputers and the DMSS disk array systems in DMSS; a detailed comparison (both in performance and functionality) between the DMF and DMSS systems LaRC's enhancements to the NSL/UniTree system administration environment the mechanism for DMSS to provide file server redundancy the statistics on the availability of DMSS the design and experiences with the locally developed transparent transition software which allowed us to make over 1.5 million DMF files available to NSL/UniTree with minimal system outage

Source record↗
DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗
NASA NTRS1994

Performance Analysis of the Unitree Central File

This report consists of two parts. The first part briefly comments on the documentation status of two major systems at NASA#s Center for Computational Sciences, specifically the Cray C98 and the Convex C3830. The second part describes the work done on improving the performance of file transfers between the Unitree Mass Storage System running on the Convex file server and the users workstations distributed over a large georgraphic area.

Pentakalos, Odysseas I.↗
NASA NTRS1985

A software toolbox for robotics

A system of programs was developed to simulate the concurrent command/response interaction between a parallel jaw end effector and the LISP program controlling it. The overall structure of the simulation system was described in a paper submitted to the IEEE Conference on Automation and Robotics. A user's guide for the system was written. A line numbering program on the VAX (Pascal), a program for aiding in file transfer from the VAX to an LSI11 over the RTNET (FORTRAN), and a file scanning program (a crude SCAN) for the LSI11 (FORTRAN) were also developed.

Bynum, W. L.↗
NASA NTRS2005

An Agent-Based Interface to Terrestrial Ecological Forecasting

The latest generation of NASA Earth Observing System (EOS) satellites has brought a new dimension to continuous monitoring of the living part of the Earth System, the biosphere. EOS data can now provide weekly global measures of vegetation productivity and ocean chlorophyll, and many related biophysical factors such as land cover changes or snowmelt rates. However, the highest economic value would come from forecasting impending conditions of the biosphere, to allow decision makers to mitigate dangers or exploit positive trends. NASA's strategic plan for the Earth Science Enterprise i d e n a s ecological forecasting as a focus for research. Ecological forecasting predicts the effects of changes in the physical, chemical and biological environment on ecosystem activity. Possible applications of such a system include predicting shortfalls or bumper crops of agricultural production, populations of threatened or invasive species or wildfire danger in time to allow improves preparation and logistical efficiency. Petabytes of remote sensing data are now available to help measure, understand and forecast changes in the Earth system, but using these data effectively can be surprisingly hard. The volume and variety of data files and formats are daunting. Simple data management activities, such as locating and transferring files, changing file formats, gridding point data, and scaling and reprojecting gridded data, can consume far more personnel time and resources than the actual data analysis. Some scientists commit to a particular data source or resolution just because using anything different would be more effort that it's worth. Better tools can help, but most of the tools developed to date are little more than shell scripts; they lack the flexibility to meet the diverse needs of users and are difficult to extend to handle changes in available data sources.

Golden, Keith↗