Search NASA⌕ Search

SEARCH · Search NASA

Results for “File Transfer System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2013

Mission Operations Center (MOC) - Precipitation Processing System (PPS) Interface Software System (MPISS)

MPISS is an automatic file transfer system that implements a combination of standard and mission-unique transfer protocols required by the Global Precipitation Measurement Mission (GPM) Precipitation Processing System (PPS) to control the flow of data between the MOC and the PPS. The primary features of MPISS are file transfers (both with and without PPS specific protocols), logging of file transfer and system events to local files and a standard messaging bus, short term storage of data files to facilitate retransmissions, and generation of file transfer accounting reports. The system includes a graphical user interface (GUI) to control the system, allow manual operations, and to display events in real time. The PPS specific protocols are an enhanced version of those that were developed for the Tropical Rainfall Measuring Mission (TRMM). All file transfers between the MOC and the PPS use the SSH File Transfer Protocol (SFTP). For reports and data files generated within the MOC, no additional protocols are used when transferring files to the PPS. For observatory data files, an additional handshaking protocol of data notices and data receipts is used. MPISS generates and sends to the PPS data notices containing data start and stop times along with a checksum for the file for each observatory data file transmitted. MPISS retrieves the PPS generated data receipts that indicate the success or failure of the PPS to ingest the data file and/or notice. MPISS retransmits the appropriate files as indicated in the receipt when required. MPISS also automatically retrieves files from the PPS. The unique feature of this software is the use of both standard and PPS specific protocols in parallel. The advantage of this capability is that it supports users that require the PPS protocol as well as those that do not require it. The system is highly configurable to accommodate the needs of future users.

Ferrara, Jeffrey↗
DOE OSTI2024

Complete and Correct Transfer of Information (CACTI)

Many distributed systems, file transfer mechanisms, and message passing systems offer reliability mechanisms such as acknowledgements, retries, and durability. While these tools may be “good enough” for their typical use cases, they may not offer sufficient coverage for the wide range of faults that impact data transfers and communication. A gap in the reliability measures may lead to some small amount of data loss. Some high-consequence systems cannot tolerate the loss or corruption of even a single record. We present seven principles that will counter a wide range of faults and protect against data loss and corruption. These principles bring together lessons learned from a wide range of technologies and can inform appropriate system design and application usage. These principles will help readers reason on how prevent data loss in a multi-hop pipeline and how to properly use tools that may have a deficiency in reliability.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2005

Fail-over file transfer process

The present invention provides a fail-over file transfer process to handle data file transfer when the transfer is unsuccessful in order to avoid unnecessary network congestion and enhance reliability in an automated data file transfer system. If a file cannot be delivered after attempting to send the file to a receiver up to a preset number of times, and the receiver has indicated the availability of other backup receiving locations, then the file delivery is automatically attempted to one of the backup receiving locations up to the preset number of times. Failure of the file transfer to one of the backup receiving locations results in a failure notification being sent to the receiver, and the receiver may retrieve the file from the location indicated in the failure notification when ready.

Semancik, Susan K.↗
NASA NTRS1984

Asynchronous file transfer to IBM PC's

The Asynchronous File Transfer System is used for interactively selecting and tramsmitting text files from the Langley Research Center's Business Data Systems Division (BDSD) host processor to an IBM Personal Computer. The IBM asynchronous communications support package is used for handling the communications on the personal computer. An application program (NATURAL, COBOL, etc.) is used for selecting and formatting the records to be transmitted, and three subroutine modules residing on the BDSD host processor are used for interfacing the application program and the communications software. Each record transmitted to the Personal Computer must be in the standard ASCII format. This record format is directly accessible by BASIC and many of the personal computer software packages provide utility programs for converting them to the format required for the particular package in question.

Hoerger, J.↗
DOE OSTI2020

Data Transfer Tool (HPC Annual Report V.3.1)

Sandia has been developing and supporting data transfer tools for over 20 years and has the expertise to take DOE into the Extreme Scale era. In looking at Exascale and beyond (Extreme Scale Computing), data sets can be thousands of 500TBs in size, a single file can be in the 100TB range, and billions of files are expected. Huge bursts of data need to be transferred, even today. While data archiving is often not thought about, it is an integral part of the full data management path when data is generated on HPC systems. In order to move generated data to its final resting place (data archive) or to transfer between file systems, a capable data transfer tool is required.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Addressing the Tension Between Strong Perimeter Control an Usability

This paper describes a strong perimeter control system for a general purpose processing system, with the perimeter control system taking significant steps to address usability issues, thus mitigating the tension between strong perimeter protection and usability. A secure front end enforces two-factor authentication for all interactive access to an enclave that contains a large supercomputer and various associated systems, with each requiring their own authentication. Usability is addressed through a design in which the user has to perform two-factor authentication at the secure front end in order to gain access to the enclave, while an agent transparently performs public key authentication as needed to authenticate to specific systems within the enclave. The paper then describes a proxy system that allows users to transfer files into the enclave under script control, when the user is not present to perform two-factor authentication. This uses a pre-authorization approach based on public key technology, which is still strongly tied to both two-factor authentication and strict control over where files can be transferred on the target system. Finally the paper describes an approach to support network applications and systems such as grids or parallel file transfer protocols that require the use of many ports through the perimeter. The paper describes a least privilege approach that dynamically opens ports on a host-specific, if-authorized, as-needed, just-in-time basis.

Hinke, Thomas H.↗
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2021

Truncated ARQ Statistical Link Analysis for Dynamic Links

The future deep space links are migrating towards higher frequency bands such as Ka band and optical. These links are susceptible to non Gaussian and non linear effects such as atmospheric turbulence, scintillation, antenna mis-pointing, jitter, etc. These dynamic links thus will experience various degrees of fading loss, and some of these link disruptions cannot be effectively mitigated by forward error correction coding and/or interleaving. One effective way to ensure reliable communication is by using Automatic Repeat Request (ARQ) protocol, where the receiver acknowledges to the transmitter whether or not a data unit is successfully received. If a data unit is not successfully received (such as after a pre-set time-out), the transmitter would then re-transmit the lost data unit to the receiver. In a previous paper, we derived a statistical link analysis method of finding the optimal operating Signal-to-Noise Ratio (SNR) and estimating the latency of an ARQ scheme. In a more recent paper, we demonstrated the above method using the SNR distribution constructed from the Ka-band (32 GHz) flight data. To simplify the discussion, we considered the academic approach that the ARQ scheme allows for an infinite number of retransmissions. In this paper, we consider the more practical case of a truncated ARQ scheme, where there is a limit on the number of retransmissions. We derive the error probability, the optimal SNR setting, and the latency statistics of the correctly received frames of the truncated ARQ schemes. We first discuss the truncated ARQ link analysis principles using the Gaussian assumption for SNR distribution with a large variance. Next, we demonstrate the statistical truncated ARQ link analysis using the SNR distribution constructed from the Ka-band flight data. The results in this paper can be applied in the design of reliable communication systems such as the Consultative Committee for Space Data System (CCSDS) File Transfer Protocol (CFTP) and the Delay Tolerant Network (DTN).

Morabito, David↗
NASA NTRSDate not supplied

Coupled Reactor and Engine Nuclear Thermal Propulsion Modeling Methodology

The design and development process of a Nuclear Thermal Propulsion (NTP) system requires extensive multiphysics modeling to couple the neutron physics and thermal feedback effects to determine the reactor’s power shape. Propulsion system performance codes utilize this power shape to determine NTP key performance parameters. While the power shape is heavily dependent on the temperature profile and geometry of the reactor, many analyses either assume a constant power shape, or use neutronics analysis to determine a power shape for a specific reactor configuration. The development of a coupling interface for a propulsion system performance code and a Monte Carlo neutron transport code (OpenMC) allows for the reactor power shape to be calculated in an iteration loop. The interface utilizes a file share system to transfer geometry dimensions, temperatures, and material identifiers to OpenMC, which is used to perform a neutron transport simulation of a design like the government Testing Reference Design reactor. The interface is then able to post-process the results from OpenMC and use the same file share system to share a power shape and other important neutron transport parameters to the system performance code. Initial results show that neglecting the changes to power shape when comparing reactor configurations can yield inaccurate results. Furthermore, utilizing propellants other than hydrogen gas can cause significant changes to the power shape, and thus, the thermal performance of a specific reactor design. This methodology is being expanded to allow for multiple families of NTP reactors to be analyzed, including block moderator, particle bed, and NERVA-derived reactors.

multiphysics coupling↗
NASA NTRSDate not supplied

Coupled Reactor and Engine Nuclear Thermal Propulsion Modeling Methodology

The design and development process of a Nuclear Thermal Propulsion (NTP) system requires extensive multiphysics modeling to couple the neutron physics and thermal feedback effects to determine the reactor’s power shape. Propulsion system performance codes utilize this power shape to determine NTP key performance parameters. While the power shape is heavily dependent on the temperature profile and geometry of the reactor, many analyses either assume a constant power shape, or use neutronics analysis to determine a power shape for a specific reactor configuration. The development of a coupling interface for a propulsion system performance code and a Monte Carlo neutron transport code (OpenMC) allows for the reactor power shape to be calculated in an iteration loop. The interface utilizes a file share system to transfer geometry dimensions, temperatures, and material identifiers to OpenMC, which is used to perform a neutron transport simulation of a design like the government Testing Reference Design reactor. The interface is then able to post-process the results from OpenMC and use the same file share system to share a power shape and other important neutron transport parameters to the system performance code. Initial results show that neglecting the changes to power shape when comparing reactor configurations can yield inaccurate results. Furthermore, utilizing propellants other than hydrogen gas can cause significant changes to the power shape, and thus, the thermal performance of a specific reactor design. This methodology is being expanded to allow for multiple families of NTP reactors to be analyzed, including block moderator, particle bed, and NERVA-derived reactors.

multiphysics coupling↗
NASA NTRS2019

Medical Data Architecture (MDA) Project Status

The Medical Data Architecture (MDA) project supports the Exploration Medical Capability (ExMC) risk to minimize or reduce the risk of adverse health outcomes and decrements in performance due to in-flight medical capabilities on human exploration missions. To mitigate this risk, the ExMC MDA project addresses the technical limitations identified in ExMC Gap Med 07: We do not have the capability to comprehensively process medically-relevant information to support medical operations during exploration missions. This gap identifies that the current in-flight medical data management includes a combination of data collection and distribution methods that are minimally integrated with on-board medical devices and systems. Furthermore, there are a variety of data sources and methods of data collection. For an exploration mission, the seamless management of such data will enable a more medically autonomous crew than the current paradigm. The medical system requirements are being developed in parallel with the exploration mission architecture and vehicle design. ExMC has recognized that in order to make informed decisions about a medical data architecture framework, current methods for medical data management must not only be understood, but an architecture must also be identified that provides the crew with actionable insight to medical conditions. This medical data architecture will provide the necessary functionality to address the challenges of executing a self-contained medical system that approaches crew health care delivery without assistance from ground support. Hence, the products supported by current prototype development will directly inform exploration medical system requirements.In fiscal year 2018, the MDA project developed Test Bed 2, the second iteration in a series of prototypes with functionality focused on data security through role-based access control and encryption, integration with One Portal exercise software and ingestion of an ultrasound Digital Imaging and Communications in Medicine (DICOM) file and image display. Test Bed 2 advances the medical data system architecture framework by providing these functionalities in a scalable system that maintained a layered, modular design. The architecture framework uses a data services approach with role-based access to data in a customized medical record system suitable for space exploration. These functionalities were demonstrated as part of the Next Space Technologies for Exploration Partnerships (NextSTEP) ground test demonstrated at the NASA Johnson Space Center Integrated Power, Avionics and Software (iPAS) facility. Interfacing to a Core Flight Software (CFS) system, the MDA system, using Consultative Committee for Space Data Systems (CCSDS) protocol, transferred an exercise file from the simulated flight MDA system to a mirrored MDA system on the ground through the CFS system. The selection of data sources and demonstrations enabled the team to address stakeholder concerns throughout the development process. In the next iteration, the MDA team will work with stakeholders to identify additional relevant functionalities to further advance system data models, standards and principles that will inform the medical system requirements development.

medical data architecture↗
NASA NTRS1991

The Jade File System

File systems have long been the most important and most widely used form of shared permanent storage. File systems in traditional time-sharing systems, such as Unix, support a coherent sharing model for multiple users. Distributed file systems implement this sharing model in local area networks. However, most distributed file systems fail to scale from local area networks to an internet. Four characteristics of scalability were recognized: size, wide area, autonomy, and heterogeneity. Owing to size and wide area, techniques such as broadcasting, central control, and central resources, which are widely adopted by local area network file systems, are not adequate for an internet file system. An internet file system must also support the notion of autonomy because an internet is made up by a collection of independent organizations. Finally, heterogeneity is the nature of an internet file system, not only because of its size, but also because of the autonomy of the organizations in an internet. The Jade File System, which provides a uniform way to name and access files in the internet environment, is presented. Jade is a logical system that integrates a heterogeneous collection of existing file systems, where heterogeneous means that the underlying file systems support different file access protocols. Because of autonomy, Jade is designed under the restriction that the underlying file systems may not be modified. In order to avoid the complexity of maintaining an internet-wide, global name space, Jade permits each user to define a private name space. In Jade's design, we pay careful attention to avoiding unnecessary network messages between clients and file servers in order to achieve acceptable performance. Jade's name space supports two novel features: (1) it allows multiple file systems to be mounted under one direction; and (2) it permits one logical name space to mount other logical name spaces. A prototype of Jade was implemented to examine and validate its design. The prototype consists of interfaces to the Unix File System, the Sun Network File System, and the File Transfer Protocol.

Rao, Herman Chung-Hwa↗
NASA NTRS1998

On-Board File Management and Its Application in Flight Operations

In this paper, the author presents the minimum functions required for an on-board file management system. We explore file manipulation processes and demonstrate how the file transfer along with the file management system will be utilized to support flight operations and data delivery.

on-board file management flight operations flight ↗
NASA NTRS1994

Performance Analysis of the Unitree Central File

This report consists of two parts. The first part briefly comments on the documentation status of two major systems at NASA#s Center for Computational Sciences, specifically the Cray C98 and the Convex C3830. The second part describes the work done on improving the performance of file transfers between the Unitree Mass Storage System running on the Convex file server and the users workstations distributed over a large georgraphic area.

Pentakalos, Odysseas I.↗
NASA NTRS2004

Software Implements a Space-Mission File-Transfer Protocol

CFDP is a computer program that implements the CCSDS (Consultative Committee for Space Data Systems) File Delivery Protocol, which is an international standard for automatic, reliable transfers of files of data between locations on Earth and in outer space. CFDP administers concurrent file transfers in both directions, delivery of data out of transmission order, reliable and unreliable transmission modes, and automatic retransmission of lost or corrupted data by use of one or more of several lost-segment-detection modes. The program also implements several data-integrity measures, including file checksums and optional cyclic redundancy checks for each protocol data unit. The metadata accompanying each file can include messages to users application programs and commands for operating on remote file systems.

Rundstrom, Kathleen↗
DOE OSTI2023

Usability and Optimization Improvements in MOOSE

The Multiphysics Object-Oriented Simulation Environment (MOOSE) framework is a foundational capability used by the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program to create over 15 different simulation tools for advanced nuclear reactors. Due to MOOSE’s broad use, improvements to the framework in support of modeling and simulation goals are critical to the program. Such improvements can take many forms, including optimization, improved user experience, streamlined application programming interfaces (APIs), parallelism, and new capabilities. The work described in this report was conducted in direct support of the simulation tools and has already been deployed. The capabilities were implemented in the same order as they are covered in this report: multiple nonlinear systems in the same input file, implementation of generic field transfers and other transfer system enhancements, support for stateful material property redistribution for adaptivity and distributed meshes, and dynamic linking and loading of individually compiled applications. These four additions are fundamental capabilities that will be leveraged by many NEAMS applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. Here, we identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

97 MATHEMATICS AND COMPUTING↗